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ASSESSING  JUDGMENT  PROEICIENCY  IN  ARMY  PERSONNEE 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  Army’s  success  in  the  current  conflicts  hinges  on  its  ability  to  maintain  tactical  and 
technical  control  in  an  operational  environment  that  demands  agility  in  mission  analysis, 
planning,  and  execution.  Because  of  the  unpredictable  nature  of  these  conflicts,  judgment  and 
decision-making  have  emerged  among  the  most  important  concepts  included  in  new  Army 
doctrine  and  as  key  findings  in  studies  that  sought  to  understand  the  skills  with  which  leaders  and 
Soldiers  must  become  proficient.  Eor  example,  a  study  in  2005  by  the  Institute  for  Defense 
Analysis  (IDA)  suggested  that,  in  general,  the  Department  of  Defense  (DoD)  does  well  at 
training  Soldiers  to  fight  traditional,  symmetric  threats  that  dominated  the  conflicts  of  the  20th 
century.  The  authors  found  less  evidence  that  current  Army  training  doctrine  and  practices 
supported  the  skills  necessary  to  deal  with  the  asymmetric  threats  of  the  21st  century.  While  all 
the  services  have  adopted  the  belief  that  judgment  and  decision-making  are  important  to  success, 
there  is  no  consensus  among  or  within  services  about  how  they  should  be  operationally  defined 
or  universally  applied. 

There  is  a  need  for  a  systematic  theoretical  analysis  of  judgment  proficiency  as  a 
military-specific  component  and  determinant  of  agility  and  the  cognitive  processes  associated 
with  it  (i.e.,  decision-making,  choice  rationale,  etc.).  Once  established,  a  conceptual  and 
theoretical  foundation  of  judgment  proficiency  can  provide  a  sound  basis  for  gaining  and 
advancing  an  understanding  of  and  informing  decisions  about  training  this  indicator  of  proficient 
agility  in  practice.  There  also  is  a  need  to  develop  reliable  and  valid  measures  of  judgment 
proficiency  that  will  apply  to  multiple  Soldier  populations  and  that  are  guided  and  refined  by  a 
theoretical  framework  and  subsequent  analyses. 

The  purpose  of  this  research  was  to  construct  a  sound  theoretical  basis  for  judgment 
proficiency  and  the  cognitive  processes  involved  in  its  application.  A  second  purpose  was  to 
develop  a  theory-based  measure  of  the  judgment  proficiency  of  Special  Eorces  (SE)  and  non-SE 
Army  Soldiers  using  a  situation  judgment  test  (SJT)  and  multiple  response  type  approach. 

Procedure: 

The  literature  review  was  based  on  the  idea  that  while  Soldiers  are  limited  by  their 
capacity  to  search  for  and  process  information,  they  are  nevertheless  cognitively  adaptive  to  the 
task  environment.  This  theme  was  central  to  how  MJP  was  conceived  in  this  research,  and  was 
discussed  in  relation  to  several  models  of  judgment  and  decision-making,  including  rational 
models,  the  Heuristics  and  Biases  (HB)  program  initiated  by  Kahneman  and  Tversky, 
Hammond’s  Cognitive  Continuum  Theory  (CCT),  the  Adaptive  Toolbox  (AT)  approach 
proposed  by  Gigerenzer  and  colleagues,  and  Klein  and  colleagues  exploration  of  Naturalistic 
Decision-Making  (NDM). 
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The  MJP  measure  was  developed  aeeording  to  the  SJT  approaeh.  Subjeet-matter  experts 
from  Speeial  Operations  Forees  (SOF)  provided  operational  stories,  lessons  learned,  multiple 
response  options  and  ratings,  and  eontinued  guidanee  during  development  of  the  test.  Follow-on 
researeh  will  eonsist  of  MJP  test  validation  researeh  with  SOF  and  non-SOF  Soldier  samples. 

Findings: 

In  our  view,  based  on  a  eomprehensive  literature  review.  Soldiers  make  effective 
decisions  by  appropriately  framing  the  situation  and  recognizing  relevant  situational  factors, 
which  enables  them  to  select  a  course  of  action  that  is  most  likely  to  result  in  their  desired 
outcome.  In  novel,  ambiguous,  rapidly  changing  situations  in  which  there  is  limited  time, 
information,  and  resources,  MJP  is  primarily  intuitive,  in  that  it  relies  on  domain-specific, 
adaptive  heuristics,  and  well-organized  knowledge  to  arrive  at  judgments  quickly.  If  time  and 
circumstances  permit,  this  initial  intuitive  approach  can  be  endorsed,  corrected,  or  overridden  by 
a  more  deliberate,  analytical  approach. 

Utilization  and  Dissemination  of  Findings: 

The  results  of  this  research  will  support  SOF  selection  and  assessment,  and  will  serve  as  a 
sound  theoretical  foundation  for  developing  Army-wide  MJP  training  and  testing.  Findings  were 
discussed  with  leaders  at  the  Special  Forces  Assessment  and  Selection  (SFAS)  Battalion  and  the 
Command  Group  from  the  Directorate  of  Special  Operations  Proponency  (DSOP). 
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ASSESSING  JUDGMENT  PROEICIENCY  IN  ARMY  PERSONNEE 


Introduction 

The  current  strategic  environment  is  characterized  by  geopolitical  uncertainties  and 
evolving  threats.  The  U.S.  Army’s  superiority  in  conventional  operations  has  led  adversaries  to 
avoid  direct  confrontation  and  instead  pursue  indireet  and  irregular  means  to  achieve  their 
objectives  (U.S.  Army,  2008).  The  U.S.  military  is  engaged  in  a  number  of  conflicts  around  the 
world  that  are  characterized  by  an  enemy  that  is  continuously  adapting  and  engaging  U.S.  forees 
using  these  irregular  means.  Maintaining  dominance  over  these  asymmetric  threats  requires  agile 
Soldiers  who  can  maintain  tactical  and  technical  control  in  environments  that  present  novel, 
ambiguous,  and  rapidly  ehanging  situations. 

The  dynamic  and  ambiguous  nature  of  these  operating  environments  poses  a  diffieult 
ehallenge  for  Soldiers  who  must  make  decisions  in  situations  characterized  by  cultural,  ethical, 
and  tactical  complexity,  often  with  limited  time,  information,  and  resources.  As  the  Army  field 
manual  on  counterinsurgency  (COIN)  states  (U.S.  Army,  2006)  “The  side  that  learns  faster  and 
adapts  more  rapidly. .  .usually  wins.”  (p.  ix). 

Sueceeding  in  these  types  of  adaptive  performance  situations  requires  a  variety  of 
knowledge,  skills,  abilities  (KSAs),  and  other  attributes  (e.g.,  see  Mueller-Hanson,  Wiseearver, 
Dorsey,  Eerro,  &  Mendini,  2009).  Some  of  the  KSAs  related  to  agility,  such  as  general 
intelligenee  or  openness,  ean  be  assessed  with  standard  assessment  tools  (see  EePine,  Colquitt,  & 
Erez,  2000;  Pulakos,  Arad,  Donovan,  &  Plamondon  2000;  Zaccaro,  2001).  Other  KSAs  such  as 
problem-solving  and  deeision-making  are  more  difficult  to  assess  (see  Endsley  &  Robertson, 
2000;  Klein,  1997;  Mueller-Hanson,  White,  Dorsey,  &  Pulakos,  2005;  Shadrick,  Crabb,  Eussier, 
&  Burke,  2007;  White  et  ah,  2005).  In  order  to  identify  and  develop  Soldiers  who  will 
demonstrate  a  pattern  of  agile  performanee  in  these  situations,  we  must  define  and  understand 
these  eomplex  skills. 

Our  goal  in  this  report  is  twofold.  The  first  is  to  define  and  describe  a  skill  we  will  eall 
“military  judgment  proficiency”  (MJP),  which  is  military  judgment  that  drives  decision-making 
in  environments  characterized  by  cultural,  legal/ethical,  and  tactical  complexity.  We  consider 
MJP  an  important  eomponent  of  adaptive  performance.  The  second  is  to  describe  the 
development  of  an  assessment  tool  that  will  distinguish  among  Soldiers  on  this  construct.  Being 
able  to  assess  MJP  can  aid  in  the  selection  and  assessment  of  Soldiers  in  specialized  units  or 
teams,  such  as  Provincial  Reconstruction  Teams  or  Special  Operations  units,  and  ean  serve  as  the 
basis  for  the  development  of  training  that  targets  MJP. 

Based  on  relevant  pieces  of  the  existing  judgment  and  decision-making  literature,  we 
conceive  of  MJP  as  a  complex  skill  that  involves:  (a)  accurately  framing  decision  problems,  (b) 
deteeting  and  interpreting  important  situational  factors  and  cues,  (c)  forecasting  the 
eonsequences  of  different  courses  of  aetion,  and  (d)  choosing  an  effective  course  of  aetion  based 
on  the  preceding  three  faetors.  As  sueh,  in  ambiguous,  novel,  rapidly  ehanging  situations  in 
whieh  there  is  limited  time,  information,  and  resourees.  Soldiers  demonstrating  MJP  are  more 
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likely  to  select  an  effective  course  of  action  by  appropriately  identifying  the  nature  of  the 
situation,  recognizing  relevant  situational  factors,  and  forecasting  the  best  overall  outcome(s) 
from  the  selected  course  of  action,  given  the  situation.  In  combination,  these  four  components  of 
MJP  enable  Soldiers  to  achieve  the  best  possible  end-state  when  faced  with  different  situations. 
After  describing  the  theory  behind  our  conceptualization  of  MJP,  we  will  describe  the 
development  of  a  measure  that  uses  multiple  formats  to  tap  each  of  the  components  of  the 
construct. 

The  first  section  of  this  report  reviews  relevant  literature  in  the  areas  of  judgment, 
decision-making,  and  problem-solving.  This  literature  review  presents  the  theoretical  foundation 
for  the  MJP  construct.  In  the  second  section  of  this  report  we  will  synthesize  the  key  themes 
from  the  literature  review  that  create  the  components  of  MJP.  Finally,  in  the  third  section  of  the 
report,  we  describe  the  development  of  a  measure  of  MJP  using  a  situational  judgment  test 
format,  and  detail  the  links  between  the  theoretical  components  and  the  measurement 
methodology. 


Understanding  Judgment  and  Decision-Making 

Military  personnel  routinely  face  novel,  dynamic  situations  in  which  to  make  difficult 
decisions  under  conditions  of  limited  time  and  information.  Research  on  decision-making  has 
shown  that  uncertain  situations  create  roadblocks  to  action  and  effective  decision-making 
(Lipshitz  &  Strauss,  1997).  Modeling  and  measuring  how  experienced  Soldiers  exhibit  MJP  and 
make  sound  decisions  in  these  situations  is  an  important  step  towards  advancing  the  development 
of  training  targeted  at  improving  judgment  and  decision-making  in  the  field.  In  order  to 
understand  the  critical  elements  of  expert  military  decision-making,  we  first  begin  with  a  review 
and  synthesis  of  several  related  literatures.  We  focus  primarily  on  the  study  of  judgment  and 
decision-making,  which  sheds  light  on  the  nature  of  MJP  for  a  variety  of  tasks  and  settings.  We 
also  draw  upon  relevant  theory  and  empirical  research  from  the  related  fields  of  problem-solving 
and  critical  thinking,  both  of  which  expand  our  understanding  of  expertise  and  proficiency  in  the 
decision-making  arena.  We  start  with  a  discussion  of  human  cognitive  capacity,  which  sets  a 
foundation  for  judgment  and  decision-making. 

Human  Cognitive  Capacity 

Decades  of  research  in  cognitive  psychology  have  produced  two  key  conclusions  about 
human  cognitive  capacities  (Hogarth,  1987).  We  highlight  them  here  because  they  underpin  how 
we  conceive  of  MJP.  First,  people  have  limited  information  processing  capacity,  especially  in 
relation  to  the  complexity  of  the  tasks  they  routinely  face.  As  such,  humans  cannot  be  thought  of 
as  calculators  who  make  optimal  decisions  based  on  all  available  information.  Rather,  the  human 
mind  is  more  akin  to  a  “. .  .selective,  sequential  information-processing  system  with  limited 
processing  and  memory  capacity”  (Hogarth,  1987,  p.  10).  This  finding  is  relevant  for  military 
personnel  who  must  take  decisive  action  in  highly  ambiguous  environments.  When  time  is  short 
and  the  situation  requires  the  individual  to  make  a  decision  and  take  action,  expecting  a  thorough 
rational  approach  where  different  options  are  carefully  weighed  is  neither  feasible  nor  realistic. 
Therefore,  if  the  conditions  restrict  individuals  from  taking  a  rational,  thorough  approach  in 
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those  types  of  situations,  then  it  is  important  to  understand  the  alternate  proeesses  they  use  to 
make  a  deeision. 

Although  we  indieate  that  humans  have  limited  proeessing  eapaeities,  it  would  be 
incorrect  to  think  of  them  as  ineffective  computers.  Humans  are  highly  adaptable  and  seek  to 
understand,  control,  and  master  their  environments  (Hogarth,  1987).  This  point  relates  directly  to 
the  second  conclusion  we  wish  to  highlight,  which  is  that  humans,  despite  the  aforementioned 
limitations,  are  adaptive  to  different  judgment  tasks.  In  addition  to  the  nature  of  judgment  tasks, 
the  contexts  in  which  they  are  performed  largely  determine  the  strategies  humans  use  for 
deciding.  We  are  interested  in  the  processes  Soldiers  initiate  when  making  judgments  and 
decisions  in  highly  dynamic  and  ambiguous  situations  and  in  a  short  amount  of  time. 

Broadly  speaking,  judgment  strategies  or  processes  can  be  intuitive,  analytical/rational,  or 
a  blend  of  both.  In  situations  characterized  by  the  need  to  make  a  decision  in  a  short  amount  of 
time,  a  deliberate,  rational  approach  that  includes  thorough  information  search  and  processing  is 
unrealistic  given  the  nature  of  the  situation.  Instead,  a  more  agile  approach  is  one  that  takes 
human  limitations  in  information  processing  into  account  and  relies  on  mental  shortcuts  or 
heuristics,  balanced  with  analysis  when  feasible.  Different  models  of  judgment  and  decision¬ 
making  have  been  defined  in  terms  of  these  intuitive  and  rational  processes,  and  we  discuss  them 
at  length  in  this  report. 

The  idea  that  people  are  limited  by  their  capacity  to  search  for  and  process  information, 
yet  cognitively  adaptive  to  the  task  environment,  is  a  recurrent  theme  in  this  literature  review. 
Moreover,  this  theme  is  central  to  how  we  conceive  of  MJP.  In  our  view,  based  on  a 
comprehensive  literature  review.  Soldiers  make  effective  decisions  by  appropriately  framing  the 
situation  and  recognizing  relevant  situational  factors,  which  enables  them  to  select  a  course  of 
action  that  is  most  likely  to  result  in  their  desired  outcome.  In  novel,  ambiguous,  rapidly 
changing  situations  in  which  there  is  limited  time,  information,  and  resources,  MJP  is  primarily 
intuitive,  in  that  it  relies  on  domain-specific  heuristics  and  well-organized  knowledge  to  arrive  at 
judgments  quickly.  If  time  and  circumstances  permit,  this  initial  intuitive  approach  can  be 
endorsed,  corrected,  or  overridden  by  a  more  deliberate,  analytical  approach. 

In  the  following  section  we  will  examine  further  the  distinction  between  rational  and 
intuitive  judgment.  In  doing  so  we  will  outline  four  key  approaches  to  judgment  and  decision¬ 
making,  each  of  which  expands  our  understanding  of  how  proficient  judgments  are  made  and  the 
extent  to  which  they  rely  on  different  processes. 

Rational  vs.  Intuitive  Judgment 

The  distinction  between  rational  and  intuitive  processes  falls  within  dual-processing 
accounts  of  human  behavior,  which  abound  in  cognitive  and  social  psychology  theories  (Evans, 
2008).  These  theories  all  have  in  common  the  notion  that  there  are  two  different  modes  of 
processing,  for  which  the  most  neutral  labels  available  in  the  literature  are  System  1  and  System 
2  processes  (Kahneman  &  Frederick,  2002;  Stanovich,  1999).  Evans’s  review  of  dual-process 
models  suggests  that  most  authors  agree  on  the  basic  distinction  between  processes  that  are 
unconscious,  rapid,  automatic,  and  high  capacity  (labeled  System  1),  and  those  that  are 
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conscious,  slow,  and  deliberate  (labeled  System  2).  For  our  eurrent  purposes,  we  will  use  the 
labels  “intuitive”  when  referring  to  System  1  and  “rational”  when  referring  to  System  2. 

Historieally,  models  of  judgment  and  deeision-making  have  been  defined  in  terms  of 
intuitive  and/or  rational  eognitive  proeesses  (Hastie  &  Dawes,  2001).  Rational  judgment  is 
defined  in  terms  of  a  logieal  eonelusion  drawn  from  the  available  information  that  is  supported 
by  a  mathematieal  proof.  These  models  refieet  an  unbounded  view  of  rationality  in  that  they  do 
not  aeeount  for  faetors  sueh  as  time  eonstraints  and  environmental  faetors  that  might  influenee 
the  judgment  proeess.  Rational  judgment  is  at  the  eore  of  normative  models  of  judgment  and 
deeision-making,  sueh  as  von  Neumann  and  Morgenstern’s  (1947)  Expected  Utility  (EU)  model, 
whieh  proposes  that  individuals  assess  the  expeeted  utility  of  eaeh  outeome  by  multiplying  the 
probability  that  the  outeome  will  oeeur  by  the  outeome’s  valanee  (i.e.,  attraetiveness).  This 
proeess  results  in  a  rational  deeision  sueh  that  the  outeome  with  the  highest  expeeted  utility  is 
ehosen. 


Intuitive  judgment,  on  the  other  hand,  aeeounts  for  the  observation  that  individuals  often 
make  ehoiees  that  run  eounter  to  what  is  expeeted  based  on  a  rational  analysis  of  the  information 
provided.  Intuitive  judgment  is  defined  in  terms  of  the  heuristies  (i.e.,  mental  shorteuts)  and 
eognitive  biases,  whieh  allow  individuals  to  draw  eonelusions  without  going  through  a  rational 
evaluation  proeess  (Kahneman,  Slovie,  &  Tversky,  1982).  This  does  not  suggest  that  individuals 
using  intuitive  judgment  are  irrational;  rather  these  models  refleet  a  bounded  view  of  rationality 
in  that  they  aeeount  for  eonstraints  in  the  mind  (e.g.,  limited  working  memory  eapaeity  and 
information  proeessing  eapaeity)  and  eonstraints  in  the  environment  (e.g.,  time  and  information), 
both  of  whieh  influenee  the  judgment  proeess.  Intuitive  judgment  is  at  the  eore  of  deseriptive 
models  of  deeision-making  sueh  as  Kahneman  and  Tversky’s  (1979)  Prospect  Theory,  whieh 
seeks  to  aeeount  for  the  diserepaneies  between  normative  rules  and  aetual  behavior. 

More  reeent  models  have  ineorporated  both  the  rational  and  intuitive  aspeets  of  judgment 
and  deeision-making.  Some  researehers  have  proposed  dual  proeess  (or  two  eontinua)  models 
that  distinguish  between  intuitive  and  rational  eognitive  operations  (see  Evans,  2008).  A  good 
example  of  this  approaeh  is  the  Attribute  Substitution  Model  of  Heuristic  Judgment  (ASM) 
proposed  by  Kahneman  and  Erederiek  (2005)  to  deseribe  judgment  and  deeision-making  under 
uneertainty.  This  model  represents  a  merger  between  the  intuitive  and  analytie  approaehes.  It  is 
analytie  in  that  the  eorreet  deeision  ean  be  determined  using  rules  of  probability,  and  it  is 
intuitive  in  that  it  elaborates  on  the  role  of  heuristies  in  the  deeision-making  proeess.  Other 
researehers  have  argued  that  rational  and  intuitive  judgment  represent  opposite  ends  of  a  single 
eontinuum.  This  is  exemplified  by  Hammond’s  (1996)  quasi-rationality  eoneeption  of  judgment 
that  assumes  most  judgments  and  deeisions  fall  along  the  mid-point  of  the  eontinuum,  and  henee 
are  driven  by  both  rational  and  intuitive  proeesses.  All  of  these  models  emphasize  the  need  to 
eonsider  both  rational  and  intuitive  judgment  to  fully  understand  the  judgment  and  deeision- 
making  proeesses. 

In  this  literature  review,  we  summarize  four  approaehes  to  understanding  deeision- 
making  that  aeeount  for  both  analytieal  and  intuitive  proeesses.  We  begin  by  giving  a  brief 
overview  of  rational  deeision-making  models,  highlighting  some  key  areas  in  whieh  they  fall 
short  of  adequately  aeeounting  for  judgment  profieieney  in  real  world  settings.  Next,  we  diseuss 
the  Heuristics  and  Biases  (HB)  approaeh,  focusing  on  one  model  in  partieular:  Kahneman  and 


4 


Frederick’s  (2005)  ASM.  The  third  approach  we  discuss  is  called  the  Adaptive  Toolbox  (AT), 
which  builds  on  research  by  Gigerenzer  and  colleagues  on  “fast  and  frugal”  heuristics  (e.g., 
Gigerenzer,  Todd,  &  the  ABC  Research  Group,  1999).  The  AT  model  differs  from  ASM  in  two 
key  ways.  First,  AT  focuses  on  decisions  made  in  situations  where  the  correct  answer  is  not 
clearly  defined,  while  ASM  uses  probability  theory  to  determine  the  best  decision.  Second,  the 
goal  of  the  AT  is  to  understand  how  individuals  make  decisions  in  real  life  settings,  while  the 
goal  of  ASM  is  to  understand  how  individuals  make  decisions  on  concrete  tasks  (e.g.,  deciding 
which  option  is  risky).  We  argue  that  these  differences  are  what  make  AT  more  relevant  than 
ASM  for  the  purposes  of  this  research  effort. 

Many  real  world  decision  environments  are  characterized  by  high  complexity,  ambiguity, 
time  pressure,  and  limited  information.  Understanding  these  demands  is  important  for 
understanding  both  the  possibilities  and  limitations  of  human  judgment  and  decision-making. 

The  study  of  naturalistic  judgments  and  decisions  provides  a  useful  examination  of  such 
demands  and  their  impact  on  judgment  and  decision-making.  The  fourth  and  final  approach  we 
will  discuss  is  that  of  Naturalistic  Decision-Making  (NDM)  (Lipshitz,  Klein,  Orasanu,  &  Salas, 
2001),  which  is  concerned  with  how  individuals,  particularly  experts,  make  decisions  in  real 
world  settings.  Although  NDM  does  not  explicitly  address  the  distinction  between  rational  and 
intuitive  processes,  it  does  provide  a  useful  and  intuitively-oriented  account  of  proficient 
decision  makers  in  naturalistic  task  environments.  Further,  we  describe  the  prototypical  NDM 
model,  Recognition-Primed  Decision  (RPD)  model,  which  suggests  a  two  phase  sequence  in 
which  a  typically  quick  assessment  is  followed  by  more  deliberate  evaluation,  time  permitting. 

Finally,  in  a  complex  and  uncertain  world,  it  might  be  said  that  decision  makers  are  aided 
by  a  variety  of  tools  to  help  them  make  reasoned  choices  and  offset  various  risks  and  stakes.  For 
example,  in  military  contexts  officers  utilize  the  Military  Decision-Making  Process  (MDMP)  to 
help  them  choose  the  best  course  of  action  from  the  most  feasible  courses  of  action  for  a  given 
situation.  While  useful  in  some  respects,  these  aids  are  not  without  limitations  akin  to  those  that 
constrain  human  information  processing.  Nowhere  have  these  trade-offs  been  more  apparent  than 
in  research  on  problem-solving.  In  the  final  section  of  this  literature  review  we  highlight  this 
research  because  it  helps  explicate  the  nature  of  expertise  and  what  differentiates  experts  from 
their  novice  counterparts.  The  NDM’s  focus  on  proficiency  lends  itself  to  a  discussion  of 
expertise  from  this  broader  perspective.  Hence,  our  discussion  of  NDM  will  also  include  relevant 
research  findings  from  the  problem-solving  literature. 

The  Rational  Model  of  Decision-Making 

The  two  meta-theories  of  coherence  and  correspondence  have  influenced  the  direction  of 
research  on  judgment  and  decision-making  from  its  beginnings  in  the  1940s  and  1950s. 
Coherence  meta-theory  refers  to  theories  and  models  that  are  concerned  with  the  coherence  or 
the  rationality  of  a  person’s  judgments  and  decisions.  Studies  of  rationality  seek  to  determine 
whether  behavior  is  coherent  and  conforms  to  axioms  of  probability  theory  and  decision  theory. 
People’s  deviations  from  expected-utility  theory  (i.e.,  from  a  presumed  standard  of  rationality) 
are  also  considered  the  interesting  phenomenon  from  this  perspective  (Goldstein,  2004). 
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The  notion  of  unbounded  rationality  -  the  idea  that  individuals  apply  appropriate  rules  of 
probability  and  statistics  when  making  decisions  in  all  situations  and  under  time  constraints  -  is 
central  to  coherence-oriented  attempts  to  understand  judgment  and  decision-making.  Examples 
of  such  rules  of  probability  and  statistical  reasoning  that  rational  models  assume  individuals 
employ  when  making  decisions  are  shown  in  Tables  1  and  2.  Note  that  the  rules  of  statistical 
reasoning  deal  with  the  extent  to  which  a  sample  is  representative  of  a  population.  One  of  the 
most  influential  models  is  the  Expected  Eltility  (ELI)  model  by  von  Neumann  and  Morgenstern 
(1947).  As  previously  mentioned,  the  ELI  model  assumes  that  individuals  use  mental  arithmetic 
to  determine  the  value  of  the  different  outcomes.  Eike  other  rational  models,  situational  factors 
(e.g.,  time  and  limitation  constraints;  framing  of  the  problem)  and  individual  differences  (e.g., 
affect,  need  for  cognition,  cognitive  ability)  are  not  considered  by  the  ELI  model. 


Table  1 


Key  Laws  of  Statistical  Reasoning  Assumed  in  Rational  Models  of  Decision-Making 


Law  of  Statistical  Reasoning  Definition 

Law  of  large  numbers 

Sample  representativeness 

Regression  to  the  mean 

Sample  validity 

Accurate  conclusions  are  more  likely  when  drawing  on  a  large  rather 
than  a  sample  of  observations. 

Accurate  conclusions  are  more  likely  when  the  sample  of  observations 
and  the  population  share  similar  key  characteristics. 

Accurate  conclusions  are  more  likely  when  the  individual  understands 
that  (a)  extreme  observations  are  likely  to  be  followed  by  less  extreme 
observations,  and  (b)  extreme  observation  of  one  event  is  likely  to  be 
associated  with  a  less  extreme  observation  of  an  event  that  is  slightly 
different  than  the  event  initially  observed. 

Accurate  conclusions  are  more  likely  when  the  sample  of  observations 
captured  the  characteristics  most  relevant  to  the  decision. 

Table  2 

Key  Rules  of  Probability  Assumed  in  Rational  Models  of  Decision-Making 

Rule  of  Probability 

Definition 

Base  rate 

Multiplicative  mles  for 
independent  events 
Multiplicative  rules  for 
dependent  events 

Additive  mles  for 
independent  events 

Additive  mles  for 
dependent  events 

The  number  of  times  an  event  has  occurred  in  the  past  (prior  probability). 

The  probability  that  two  independent  events  will  occur  is  equal  to  the 
product  of  their  individual  probabilities  (joint  probabilities). 

The  probability  that  two  dependent  events  will  occur  is  equal  to  the 
product  of  their  individual  probabilities  (conditional  probabilities). 

The  probability  that  at  least  one  of  a  number  of  independent  events  will  be 
observed. 

The  probability  that  at  least  one  of  a  number  of  dependent  events  will  be 
observed. 

Eipshitz  et  ah,  (2001)  identify  rational  approaches  such  as  the  ELI  model  as  having  four 
essential  characteristics:  (a)  choice  (i.e.,  decision-making  is  conceptualized  as  choosing  among 


6 


concurrently  available  alternatives),  (b)  input-output  orientation  (i.e.,  models  focus  on  predicting 
which  alternative  will,  or  should  be,  ehosen  given  a  deeision  maker’s  preferenees),  (e) 
comprehensiveness  (i.e.,  deeision-making  is  eoneeptualized  as  a  deliberate  and  analytie  proeess 
that  requires  a  relatively  thorough  information  search),  and  (d)  formalism  (i.e.,  the  development 
of  abstraet  and  eontext-free  models  amenable  to  quantitative  testing). 

Pure  rational  models  with  these  eharaeteristies  are  not  direetly  relevant  to  the  eurrent 
researeh,  so  detailed  deseriptions  are  not  provided.  A  key  shorteoming  of  rational  models  is  the 
very  notion  of  eompletely  unbounded  rationality.  Rational  models  of  judgment  and  decision¬ 
making  focus  on  making  explicit  the  ehoiees,  uneertainties,  and  outeomes  of  the  deeision 
problem  (Collyer  &  Maleeki,  1998).  Yet,  researeh  shows  that  the  informational  requirements  of 
these  models  are  too  demanding  in  eomparison  to  the  limitations  assoeiated  with  human 
cognitive  eapaeities  (Gigerenzer  &  Goldstein,  1999;  Lipshitz,  Klein,  Orasanu,  &  Salas,  2001),  as 
well  as  the  eonstraints  imposed  by  real  world  environments.  Furthermore,  the  time  it  takes  to 
eolleet  all  of  the  neeessary  information  required  by  rational  models  makes  them  ill-suited  for 
situations  requiring  immediate  action  (Collyer  &  Maleeki,  1998).  This  includes  dynamie 
situations  faeed  by  military  leaders  characterized  by  insuffleient  information,  laek  of  time,  and 
eritieality  of  outeomes  (e.g.,  life  or  death  of  subordinates).  Therefore,  where  military  judgment 
and  deeision-making  are  eoneerned,  the  elassieal  ideal  of  appropriately  applying  the  rules  of 
probability  and  statisties  is  often  both  prohibitive  and  unrealistic. 

In  sum,  for  real  world  situations  eharaeterized  by  eomplexity,  novelty,  ambiguity  of 
information  (e.g.,  not  knowing  the  aeeuraey  of  the  information  you  have  as  well  as  the  amount  of 
information  you  are  laeking),  as  well  as  time  pressure,  typical  human  information  processing 
eapaeities  simply  eannot  handle  the  rigors  imposed  by  rational  models  of  judgment  and  decision¬ 
making.  Further,  beeause  humans  laek  the  mental  eapaeity  to  optimize  and  behave  in  aeeordanee 
with  normative  models,  the  validity  of  sueh  models  as  a  suitable  deseription  of  human  judgment 
and  decision-making  is  questionable.  In  the  next  section  we  examine  an  alternative  approach, 
which  proposes  that  the  eognitive  proeesses  that  strueture  judgment  and  deeision-making  do  not 
eoineide  with  normative  prineiples,  but  instead  rely  on  simplifying  mental  shorteuts,  or 
heuristics.  While  these  shorteuts  ean  lead  to  eognitive  biases,  heuristics  are  adaptive  to  the 
constraints  of  limited  information  proeessing  eapaeity  and  the  demands  of  situational  eonditions. 
As  sueh,  the  Heuristics  and  Biases  (HB)  program,  as  it  is  known,  provides  a  better  aceount  of 
people’s  systematie  deviations  from  rational  ehoiee. 

Heuristics  and  Biases  (HB)  Program 

The  dominant  program  of  judgment  and  deeision-making  researeh  sinee  the  1970s  has 
been  the  HB  approaeh  initiated  by  Daniel  Kahneman  and  Amos  Tversky  (Phillips,  Klein,  &  Seik, 
2004).  It  is  also  eonsidered  under  the  umbrella  of  the  eoherence  meta-theory  of  judgment  and 
deeision-making,  although  its  primary  eontribution  to  researeh  has  been  to  demonstrate  the  sharp 
departures  of  human  judgment  from  normative  prineiples  (rather  than  adherenee  to  them).  In 
partieular,  Tversky  and  Kahneman  (1974)  proposed  that  deeisions  are  often  made  using 
psychologieal  shortcuts  or  heuristics.  For  example,  the  representativeness  heuristie  refers  to  a 
tendency  to  make  probability  judgments  based  on  the  similarity  of  an  event  to  an  underlying 
souree;  the  greater  the  similarity,  the  higher  the  probability  estimate.  Table  3  provides  several 
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more  examples  of  heuristics  and  biases,  and  which  rule(s)  of  statistical  reasoning/probability 
they  violate. 

Table  3 


Heuristics  and  Biases  in  Judgment  and  Decision-making 


Heuristic/Bias 

Description 

Violated  Rules  of  Statistical 
Reasoning/Probability 

Anchoring  and 
adjustment 

Tendency  to  make  insufficient 
adjustments  from  an  extreme  starting 
point  (or  anchor)  when  estimating  the 
probability  of  an  event  occurring. 

Regression  to  the  mean. 

Availability 

Tendency  to  overestimate  the 
probability  of  an  event  occurring 
because  it  is  easily  retrieved  from 
memory. 

Law  of  large  numbers. 

Sample  representativeness. 

Concrete  information 

Tendency  to  give  more  weight  to 
observable  information  than  abstract 

Law  of  large  numbers. 

information  even  if  the  abstract 
information  is  more  reliable. 

Sample  representativeness. 

Dilution 

Tendency  to  underestimate  the 
probability  that  an  event  will  occur 
when  irrelevant  information  is  given 
along  with  relevant  information. 

Sample  validity. 

Gambler’s  fallacy 

Tendency  to  overestimate  the 
probability  of  the  occurrence  of  an 
event  not  previously  observed. 

Multiplicative  rules  of 
probability. 

Ignoring  relative 
frequency 

Tendency  to  make  estimates  based  on 
observed  frequencies  rather  than  on 
relative  frequencies,  ignoring  the 
statistical  independence  of  random, 
discrete  events. 

Base  rate. 

Misperceptions  about 
equiprobability 

Tendency  to  define  incorrectly  the 
probability  that  two  or  more  events  will 
occur  as  equal  when  in  fact  they  are  not. 

Additive  rules  of  probability. 

Order  effect 

Tendency  to  estimate  the  probability  of 
an  occurrence  based  on  either  the  initial 
(primacy)  or  most  recent  (recency) 
observations. 

Law  of  large  numbers. 

Sample  representativeness. 
Regression  to  the  mean. 

Overestimating 
conjunctive  events 

Tendency  to  overestimate  the 
probability  that  two  events  will  occur. 

Multiplicative  rules  of 
probability. 

Representativeness 

Tendency  to  assign  an  event  to  a 
category  because  it  resembles  other 
events  in  the  category. 

Law  of  large  numbers. 

Sample  representativeness. 
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Although  psychologically  straightforward,  heuristics  often  lead  to  biases  beeause  relevant 
information  (e.g.,  base  rates)  is  frequently  not  taken  into  aecount.  However,  onee  uneovered, 
these  biases  are  useful  in  that  they  serve  to  illuminate  the  heuristic  nature  of  the  underlying 
system  (Gilovieh  &  Griffin,  2002;  Kahneman  &  Tversky,  1972).  The  HB  approaeh  has  been 
applied  to  domains  as  diverse  as  aceounting,  management,  and  marketing,  and  helps  aceount  for 
a  host  of  suboptimal  deeisions  (Shanteau,  1992). 

Attribution  Substitution  Model  of  Heuristic  Judgment  (ASM) 

Recently,  Kahneman  and  Frederiek  (2002,  2005)  have  proposed  a  model  that 
incorporates  findings  from  earlier  work  by  Kahneman  and  colleagues  (e.g.,  Kahneman,  2003; 
Kahneman  &  Tversky,  1979)  called  the  Attribution  Substitution  Model  of  Heuristic  Judgment 
(ASM).  The  model  expands  on  the  earlier  work  by  (a)  using  the  notion  of  attribute  substitution  to 
explain  why  heuristies  work,  (b)  deseribing  the  role  of  heuristies  in  judgments  in  both  certain  and 
uneertain  situations,  and  (c)  introducing  a  process  by  whieh  judgments  based  on  heuristics  are 
modified  or  dismissed  as  inaccurate. 

In  deseribing  their  model,  Kahneman  and  Frederiek  (2005)  adopt  the  labels  System  1  and 
System  2  (see  Stanovieh  &  West,  2002)  to  distinguish  between  intuitive  and  reflective  (i.e., 
rational)  cognitive  operations  that  influence  the  judgment  proeess.  The  cognitive  processes  in 
System  1  are  fast,  automatie,  effortless,  associative,  and  diffieult  to  control.  In  contrast,  the 
cognitive  proeesses  in  System  2  are  slow,  serial,  effortful,  deduetive,  and  deliberate.  The  systems 
also  differ  in  terms  of  the  informational  content  on  which  they  act;  System  1  proeesses  aet  on 
eontent  that  is  arousing  (e.g.,  affeetive,  eoncrete,  and  speeifie),  whereas  System  2  proeesses  aet 
on  eontent  that  is  neutral  (e.g.,  involving  statisties). 

While  both  System  1  and  System  2  eontribute  to  the  final  judgment,  the  proeess  begins 
with  the  intuitive  operations  involved  in  System  1 .  This  intuitive  judgment  is  the  result  of 
automatie  proeesses  that  involve  well  developed  sehemas  and/or  produetions  (Anderson,  1990) 
that  ehunk  information  and  proeedures,  and/or  heuristies  that  allow  for  quick  judgments  based  on 
incomplete  information.  That  is,  people  use  heuristies  as  a  substitution  for  attributes  that  are  not 
observed  in  the  environment.  The  ASM  model  assumes  that  the  heuristies  lead  to  violations  of 
both  the  laws  of  statistieal  reasoning  and  the  rules  of  probability,  and  henee  reduee  rather  than 
enhance  deeision  aeeuracy.  However,  once  System  1  produces  a  response.  System  2  is  aceessed 
in  an  effort  to  determine  the  quality  of  the  intuitive  judgment.  This  evaluation  ean  lead  to  an 
endorsement,  correetion,  or  overriding  of  the  intuitive  judgment  produced  by  System  1.  Thus,  the 
ASM  model  allows  for  aeeurate  heuristic-based  judgments. 

Several  other  eharaeteristics  of  System  1  and  System  2  are  worth  noting.  First,  both 
systems  can  be  aetive  at  the  same  time.  Indeed,  the  automatie  operations  in  System  I  compete 
with  the  eontrolled  operations  in  System  2  to  influence  the  overall  response  (or  judgment).  While 
System  1  and  System  2  are  independent  in  terms  of  cognitive  processes  and  the  eontent  acted 
upon,  the  impression  generated  in  System  1  serves  as  an  anehor  for  the  deliberate  proeess  that 
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occurs  in  System  2.  Thus,  individuals  who  engage  in  deliberate  decision-making  processes  are 
unaware  that  their  final  judgment  is  infiuenced  by  the  impression  generated  at  the  unconseious 
level. 

Perhaps  the  most  relevant  aspect  of  the  model  offered  by  Kahneman  and  Frederick 
(2005)  is  that  it  identifies  the  proeesses  in  System  2  that  are  likely  to  result  in  an  effeetive 
judgment.  For  example,  judgment  is  improved  when  individuals  deliberately  generate  courses  of 
action  that  are  viable  alternatives  to  the  one  generated  by  the  intuitive  proeess  before  aeting.  As 
we  will  discuss  in  a  subsequent  section,  research  on  problem-solving  concurs  with  this 
conclusion,  although  the  extent  to  whieh  a  solution  ean  be  deliberately  and  rationally  generated 
in  naturalistic  settings  is  questionable. 

Cognitive  Continuum  Theory  (CCT) 

Before  concluding  our  discussion  of  the  ASM  model,  we  turn  briefly  to  a  seeond  model 
that  also  merges  intuitive  and  analytic  approaches.  Unlike  the  ASM,  which  developed  from  the 
perspective  of  judgment  eoherence  (i.e.,  rationality),  Hammond’s  (1996)  Cognitive  Continuum 
Theory  (CCT)  developed  out  of  the  second  meta-theory  of  judgment  and  deeision-making  called 
“Correspondence.”  Correspondence  meta-theory  refers  to  theories  that  are  concerned  with  the 
correspondence  of  judgment  with  empirieal  facts.  In  other  words,  it  focuses  on  the  empirical 
accuracy  of  a  judgment  rather  than  its  coherence  or  rationality.  The  CCT  is  an  extension  of 
Social  Judgment  Theory,  whieh  is  primarily  eoncerned  with  understanding  how  experts  form 
judgments  based  on  cues  provided  by  the  environment.  Accuraey  of  judgment  from  the  CCT 
perspective  depends  on  the  extent  to  which  an  individual’s  mental  model  matches  the 
environment  in  terms  of  cues,  relations  between  cues  and  the  target  event,  as  well  as  the  relative 
importanee  of  cues  (Hogarth,  1987).  Social  Judgment  Theory  itself  developed  from  Brunswik’s 
(1952)  Lens  Model,  whieh  describes  the  relationships  between  the  objeetive  environment  (or 
task  system),  the  information  that  is  available  on  this  environment,  the  subjective  pereeption  and 
integration  of  this  information,  and  the  judgments  and  deeisions  to  whieh  they  lead  (Lipshitz, 
1993). 


In  a  pioneering  Psychological  Review  article,  Hammond  (1955)  showed  how  Brunswik’s 
work  on  pereeption  was  relevant  to  the  task  of  making  judgment  inferences  from  incomplete  and 
fallible  cues  in  the  environment.  More  recently,  Hammond  (1996)  proposed  CCT  and  eontrasts 
intuitive  and  analytic  thinking  by  rejecting  the  notion  that  judgments  and  decisions  must  be 
based  entirely  on  intuition  or  analysis  in  their  pure  forms.  Instead  he  suggests  that  the  cognitive 
proeesses  that  guide  judgment  and  deeision-making  ean  be  loeated  along  a  cognitive  continuum 
that  ranges  between  intuition  and  analysis.  A  proeess  is  more  intuitive  (or  less  analytical)  to  the 
extent  that  it  is  executed  under  low  control  and  low  conscious  awareness,  rapid  rate  of  data 
processing,  high  confidenee  in  answer  and  low  confidenee  in  the  method  that  produced  it. 
Hammond’s  position  is  that  judgments  and  decisions  often  involve  both  intuitive  and  analytic 
cognitive  elements,  whieh  beeome  blended  in  different  strategies.  As  such,  many  decisions  are 
located  somewhere  near  the  quasi-rational  center  of  the  cognitive  continuum  (Goldstein,  2004). 

Contrasting  ASM  and  CCT 

Compared  to  the  ASM  model,  CCT  pays  more  attention  to  the  task  environment,  making 
its  assertions  about  judgment  more  relevant  to  our  conception  of  MJP.  Speeifically,  CCT  posits 
that  certain  task  properties  are  likely  to  induce  more  intuitive  processing,  while  other  task 
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properties  are  likely  to  induee  more  analytie  proeessing.  Thus,  not  only  can  cognitive  processes 
be  arranged  on  a  cognitive  continuum  (as  more  or  less  intuitive  or  analytical),  but  so  too  can 
tasks  be  arranged  on  a  task  continuum  that  represents  the  extent  to  which  they  induce  more  or 
less  intuition  or  analysis  (Lipshitz,  1993).  Hammond,  Hamm,  Grassia,  and  Pearson  (1987) 
conducted  an  experiment  with  expert  highway  engineers  to  test  CCT  hypotheses  about  the 
interactive  effects  of  task  characteristics  and  mode  of  cognition  on  the  quality  of  performance. 
They  found  support  for  task  conditions  inducing  modes  of  cognition  with  corresponding 
properties. 

In  sum,  Hammond  suggests  that  most  real-world  decisions  are  made  in  a  quasi  rational 
manner,  blending  intuition  and  analysis.  His  CCT  model  takes  into  account  the  limited 
information  processing  capacity  that  constrains  human  cognition,  but  also  recognizes  that 
humans  are  adaptive  to  the  nature  of  the  judgment  task.  Like  the  ASM  model,  it  blends  intuitive 
and  rational  accounts  of  thinking  and  further  explicates  the  nature  and  role  of  intuition  in 
dynamic  decision  processes.  Unlike  the  ASM  model,  however,  CCT  underscores  the  importance 
of  analyzing  the  nature  of  the  decision  task  at  hand. 

In  concluding  this  section  on  the  HB  program,  we  find  that  the  ASM  model  does  provide 
a  useful  starting  point  for  understanding  the  processes  that  underlie  MJP.  It  accounts  for 
departures  from  normality  and  proposes  heuristics  as  simplified  methods  for  coping  with 
humans’  limited  information  processing  capacity.  While  acknowledging  that  such  limitations 
impact  judgment  strategy,  this  program  of  research  has  little  to  say  about  actual  task 
environments  and  how  parameters  of  naturalistic  or  field  settings  impose  additional  constraints 
on  judgment  and  decision-making.  Therefore,  at  least  two  important  aspects  of  the  ASM  model 
make  it  less  valuable  as  the  theoretical  basis  for  understanding  MJP.  First,  rather  than  being 
applicable  to  decisions  made  in  situations  where  the  correct  answer  is  not  clearly  defined,  the  HB 
approach  and  the  ASM  model  in  particular  utilize  probability  theory  to  determine  a  single  best 
decision.  Second,  the  goal  of  the  ASM  model  is  to  understand  how  individuals  make  decisions 
on  concrete  tasks  (e.g.,  deciding  which  option  is  risky)  rather  than  to  understand  how  individuals 
make  real-life  decisions  in  applied  settings. 

The  AT  approach,  discussed  next,  takes  both  these  aspects  into  consideration  and  returns 
to  the  Brunswikian  origins  of  Social  Judgment  Theory  by  emphasizing  the  adaptibility  of  an 
organism’s  behavior  to  not  only  human  limitations,  but  also  environmental  constraints 
(Goldstein,  2004).  The  approach  specifies  domain-specific  heuristics  that  are  adaptive  in  a  world 
where  decision-making  speed  and  the  ability  to  make  use  of  limited  information  are  paramount. 
As  such,  this  approach  is  more  clearly  aligned  with  the  decision-making  requirements  of  novel, 
ambiguous,  rapidly  changing  situations  and  is  more  relevant  to  understanding  judgment 
proficiency  in  the  military. 

Adaptive  Toolbox  (AT) 

The  AT  model  was  offered  by  Gigerenzer  et  ah,  (1999)  as  a  way  to  describe  the  cognitive 
strategies  available  to  individuals  when  making  decisions  in  everyday  life.  The  model  is  built  on 
Simon’s  (1956)  Satisficing  Model  that  assumes  human  judgment  is  best  understood  in  terms  of 
bounded  rationality  rather  than  the  unbounded  rationality  offered  by  earlier  models.  According  to 
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the  satisficing  model,  rationality  is  bounded  by  limitations  in  human  cognition,  and  by  the 
environmental  conditions  in  which  humans  operate.  Because  of  these  constraints,  individuals  use 
a  satisficing  heuristic  to  make  decisions,  a  process  that  ends  when  a  reasonable,  but  not 
necessarily  perfect  conclusion  is  drawn.  As  such,  people  are,  at  best,  rational  given  the 
information  they  have  and  the  limitations  of  the  human  cognitive  system.  They  do  not  consider 
every  available  option,  but  instead  choose  the  one  that  best  fits  the  situation  and  satisfies  the 
decision  maker  (Collyer  &  Malecki,  1998;  Simon,  1957).  This  means  that  judgments  and 
decisions,  as  well  as  problem-solving  in  general,  are  not  optimized  per  se,  but  satisfied.  In  some 
cases,  satisficing  leads  to  fast  decisions.  However,  it  can  also  result  in  an  extended  process  that 
involves  comparing  the  value  of  multiple  attributes  before  reaching  a  decision. 

The  AT  approach  suggests  that  judgment  and  decision-making  are  developed  over  time 
and  are  based  on  each  individual’s  past  experiences.  It  provides  the  blueprint  for  the  use  of 
heuristics  for  solving  domain  or  context  specific  problems  (Gigerenzer  et  ah,  1999).  The 
heuristics  found  in  the  AT  are  based  on  three  premises:  (a)  psychological  plausibility  (how 
people  actually  make  decisions),  (b)  domain  specificity  (specialized  heuristics  based  on  cognitive 
and  emotional  building  blocks),  and  (c)  ecological  rationality  (the  adaptation  of  heuristics  to  the 
social  and  physical  environment  in  which  they  are  used)  (Gigerenzer,  2001). 

The  third  premise,  ecological  rationality,  is  adapted  from  Simon’s  (1956)  original 
conception  of  bounded  rationality.  More  than  just  stipulating  that  humans  rely  on  heuristics  due 
to  limitations  in  time,  information,  and  computational  ability,  Simon  proposed  that  these 
heuristics  must  be  understood  in  the  context  of  the  environmental  structures  to  which  they  are 
adjusted.  The  HB  program  does  not  consider  task  environment,  which  is  the  ecological  and 
contextual  aspects  of  bounded  rationality.  In  recent  years,  ecological  rationality  has  provided  a 
successful  alternative  interpretation  of  the  idea  of  bounded  rationality. 

The  domain  specificity  characteristic  of  the  AT  model  is  encapsulated  in  the  notion  of 
fast  and  frugal  heuristics.  These  are  domain-specific  heuristics  that  are  computationally  simple 
(i.e.,  fast)  and  sparing  in  their  information  requirements  (i.e.,  frugal).  While  the  HB  approach 
often  views  heuristics  as  inferior,  limited,  and  relied  upon  to  the  detriment  of  the  decision-maker, 
Gigerenzer  and  colleagues  (e.g.,  Gigerenzer  &  Todd,  1999)  see  heuristics  as  advantageous  and 
representing  a  means  by  which  information  in  the  environment  can  be  used  to  arrive  at 
reasonable  decisions.  Fast  and  frugal  heuristics  stem  from  constraints  on  information  processing, 
a  consequence  of  which  is  that  they  make  use  of  a  limited  search.  As  such,  not  all  available 
information  is  processed,  and  only  a  fraction  of  this  information  influences  judgment 
(Gigerenzer  &  Goldstein,  1999).  This  makes  fast  and  frugal  heuristics  adaptable  in  situations  in 
which  decisions  must  be  made  quickly  and  with  limited  information,  provided  accuracy  does  not 
suffer  too  greatly  (Goldstein,  2004). 

Fast  and  frugal  heuristics  are  considered  similar  to  the  satisficing  heuristic  in  that  they  are 
bounded  by  the  environment.  They  differ  from  satisficing  in  two  ways.  First,  fast  and  frugal 
heuristics  almost  always  provide  a  quick  decision.  Second,  fast  and  frugal  heuristics  are  clearly 
tied  to  the  structure  of  the  environment  (i.e.,  ecological  rationality).  As  a  result,  the  AT  includes 
bins  that  contain  heuristics  that  can  be  used  in  specific  situations.  According  to  the  AT  model, 
individuals  may  only  need  to  employ  a  single  heuristic  when  making  simple  decisions.  However, 
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multiple  heuristics  might  be  necessary  when  making  complex  decisions.  Specifically,  individuals 
employ  heuristics  that  guide  the  search  for  information,  determine  when  to  stop  the  information 
search,  and  produce  a  final  decision. 

Many  decisions  require  individuals  to  select  between  two  or  more  alternatives  without  the 
benefit  of  sufficient  information.  In  these  instances,  individuals  are  motivated  to  search  for 
information  that  will  facilitate  the  decision-making  process.  In  some  cases,  individuals  will 
employ  a  heuristic  that  leads  to  a  simple  random  search  for  information.  In  other  cases, 
individuals  might  use  a  heuristic  that  leads  to  a  search  guided  by  a  single  attribute  that  was 
previously  useful  in  evaluating  attributes  in  similar  situations.  Regardless,  the  fast  and  frugal 
heuristics  lead  to  a  quick  information  search.  Further,  because  the  AT  model  assumes  that 
individuals  make  decisions  under  time  and  environmental  constraints,  individuals  need  to 
employ  a  criterion  for  stopping  the  information  gathering  process  in  a  reasonable  timeframe. 
When  limited  information  is  available,  the  individual  may  adopt  a  take-the-first  heuristic 
approach.  When  using  this  heuristic,  the  individual  stops  searching  for  alternatives  once  an 
example  that  includes  an  important  or  relevant  attribute  is  identified. 

In  sum,  regardless  of  whether  the  decision  is  simple  or  complex,  individuals  employ 
heuristics  to  make  decisions.  For  many  problems,  an  optimal  solution  is  not  feasible,  especially 
the  kind  home  out  of  novel,  ambiguous,  rapidly  changing  situations,  where  the  optimal  solution 
is  not  readily  apparent,  and  may  never  be.  Psychological  shortcuts  are  adaptive  to  both  the 
limited  information  processing  capacity  humans  bring  to  most  judgment  and  decision-making 
tasks  as  well  as  the  structure  of  the  environment.  Unlike  the  general  heuristics  offered  by 
Kahneman  and  his  colleagues  (e.g.,  Kahneman  &  Tversky,  1982),  fast  and  frugal  heuristics  are 
specific  to  the  context  and  take  advantage  of  environmental  constraints.  This  degree  of 
specificity,  according  to  the  AT  model,  allows  for  smart  heuristics  that  yield  effective  decisions 
(Gigerenzer,  2001).  Both  the  classical  rationality  and  HB  programs  largely  ignore  the  important 
role  played  by  the  environment  in  shaping  human  behavior.  In  contrast,  Gigerenzer  and 
colleagues  (e.g.,  Gigerenzer  &  Todd,  1999)  explicitly  acknowledge  the  task  environment  as 
constraining  the  type  of  judgment  and  decision-making  in  which  individuals  engage. 

For  our  current  purposes,  the  task  environment  involves  real  world  settings  in  which 
decision  makers  confront  novelty,  ambiguity,  and  complexity  under  conditions  of  limited  time 
and  resources,  either  cognitive,  material,  or  otherwise.  We  have  argued  in  this  review  that  it  is 
important  to  understand  judgment  proficiency  within  this  specific  context.  The  AT  model  is 
useful  in  highlighting  that  the  rationality  of  heuristics  is  not  logical,  but  ecological,  and  that  it  is 
relative  to  the  environment.  Evolved  capacities  in  humans  to  accommodate  a  limited  cognitive 
system  make  heuristics  simple;  the  structure  of  the  environment  makes  them  smart  (Gigerenzer, 
2001).  Naturalistic  Decision-Making  (NDM),  discussed  later  in  this  review,  is  particularly 
interested  in  decision-making  under  certain  conditions;  those  of  limited  time,  knowledge,  and 
other  limited  resources.  As  such,  NDM  is  related  to  bounded  rationality,  generally  (Todd  & 
Gigerenzer,  2001),  but  with  even  clearer  parallels  to  ecological  rationality,  specifically.  Both 
NDM  and  AT  focus  on  simple  decision-making  heuristics  that  are  task  specific  (versus  general 
purpose)  and  computationally  fast  and  frugal  (versus  optimizing). 

A  key  distinguishing  characteristic  of  NDM  is  its  focus  on  proficient,  expert  decision 
makers.  These  are  individuals  with  “. .  .relevant  experience  or  knowledge  in  the  decision-making 
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domain  who  rely  on  their  experienee  directly”  (Phillips  et  ah,  2004,  p.  334).  Expert  decision¬ 
making  in  real  world  contexts  is  central  to  NDM  research  and  further  illuminates  the  nature  of 
MJP.  Before  proceeding  with  a  discussion  of  the  NDM  literature,  however,  it  is  useful  to  further 
explore  the  nature  of  expertise.  In  doing  so,  we  will  review  several  key  distinctions  between 
experts  and  novices  that  have  contributed  greatly  to  research  on  judgment  and  decision-making. 
The  role  of  expertise  is  particularly  relevant  to  military  settings  because  of  the  hierarchical  nature 
of  the  military  organization.  In  the  military,  both  officers  and  enlisted  Soldiers  continuously  train 
to  build  expertise  in  their  particular  branches  and  military  occupational  specialty  (MOS)  in  order 
to  make  effective  decisions  that  will  impact  themselves,  their  unit,  and  their  overarching  mission. 
As  such,  most  Soldiers  in  positions  of  authority  have  built  a  minimum  amount  of  expertise  in 
their  MOS  before  being  put  in  situations  requiring  them  to  make  decisions  and  take  action. 

Problem-Solving  Expertise 

A  great  deal  of  what  we  know  about  expertise  has  been  identified  by  researchers  studying 
problem-solving.  During  the  1980s,  problem-solving  research  witnessed  a  shift  in  interest  from  a 
focus  on  lab-based  puzzles  to  examining  how  people  solve  problems  in  natural  knowledge 
domains.  The  so-called  transformation  problems  (e.g..  Tower  of  Hanoi)  dominant  in  lab  research 
are  relatively  simple  and  knowledge-independent.  Their  popularity  can  be  attributed  to  the  fact 
that  they  allowed  researchers  to  easily  trace  participants’  problem-solving  steps  and  identify 
whether  heuristics  were  used  in  the  problem  solution.  Yet,  these  steps  and  heuristics  only  provide 
limited  insight  into  the  kind  of  problem-solving  required  in  natural  knowledge  domains.  As  such, 
researchers  shifted  their  focus  to  problems  in  more  varied  and  applied  areas,  such  as  physics, 
writing,  and  chess  playing. 

Perhaps  an  inevitable  extension  of  this  research  was  an  interest  in  comparing  the 
performance  of  experts  and  novices  in  specific  domains,  and  accounting  for  any  observed 
differences.  Insights  about  problem-solving  from  this  research  have  been  important  to  our 
emerging  understanding  of  how  people  solve  problems  as  well  as  make  judgments  and  decisions. 
In  this  section  we  examine  in  more  detail  the  important  role  of  knowledge  in  the  demonstration 
of  expertise,  as  well  as  the  core  processes  that  underlie  both  novice  and  expert  problem-solving. 
We  also  take  problem  type  into  consideration  and  explore  expertise  in  the  context  of  the  specific 
kinds  of  situations  on  which  the  current  project  is  focused. 

Importance  of  Knowledge 

A  well-documented  finding  is  that  experts  solve  problems  differently  from  novices.  In 
research  comparing  how  experts  and  novices  engage  in  problem-solving,  one  of  the  most 
predictable  differences  between  the  two  groups  is  the  amount  of  domain-specific  knowledge 
possessed  by  each  (Ericsson  &  Hastie,  1994).  A  general  conclusion  from  expert-novice 
comparison  research  is  that  when  people  are  dealing  with  familiar  material  they  rely  to  a  greater 
extent  on  previously  memorized  schemata,  which  are  abstract  mental  structures  of  organized 
knowledge.  Chase  and  Simon  (1973)  showed  that  experts  do  not  apply  different  or  superior 
processes  to  solving  problems,  but  their  better  performance  can  be  attributed  to  the  nature, 
amount,  and  organization  of  knowledge  they  possess  about  a  particular  domain.  In  other  words. 
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experts  are  able  to  apply  previously  learned  knowledge  and  rules  that  led  to  a  solution  in  the  past 
(Hunt,  1994). 

Schemata  exist  in  networks  and  represent  each  individual’s  unique  understanding  of  the 
world.  Experts  differ  from  novices  in  having  large,  rich,  well-organized  schemata  comprised  of 
declarative  or  factual  knowledge  about  a  particular  domain.  Problem-solving  and  the  encoding 
and  retrieval  of  information  from  long-term  memory  appear  to  be  schemata  driven  (Lipshitz, 
Levy,  &  Ore  hen,  2006).  By  applying  these  schemata,  experts  are  able  to  develop  sophisticated, 
albeit  abstract,  representations  of  problems  based  on  structural  similarities  among  problems 
(Sternberg,  2000).  Indeed,  much  of  expert  problem-solving  depends  on  the  expert’s  ability  to 
learn  how  to  describe  the  external  world  so  that  descriptions  can  be  related  to  the  abstract 
representation  in  the  expert’s  memory.  As  such,  schema  application  often  depends  on  abstract, 
sophisticated  analyses  of  situations  (Hunt,  1994). 

Recall  that  the  intuitive  judgment  and  corresponding  processes  underlying  Kahneman  and 
Frederick’s  (2005)  notion  of  System  1  involves  well-developed  schemata.  Because  of  reliance  on 
schemata,  problem-solving  for  experts  is  not  nearly  as  effortful  as  it  is  for  novices,  who  lack 
organized  knowledge  about  the  domain  in  question.  Experts  can  quickly  organize  new 
perceptions  into  existing  schemata  and  act  with  much  less  effort.  Schemata,  therefore,  provide  an 
orderly  way  to  shift  attention  from  one  aspect  of  a  problem  to  another.  When  applied  correctly, 
they  transfer  the  information  processing  burden  from  working  memory,  where  the  human 
problem-solver  is  weak,  to  long-term  memory,  where  the  problem-solver  is  relatively  strong 
(Hunt,  1994). 

Limits  on  working  memory  determine  how  complicated  the  representation  of  the  problem 
can  be  (Hunt,  1994).  A  key  finding  in  comparing  experts  and  novices  is  that  the  former  are  able 
to  recall  more  information  about  a  problem  than  novices  when  the  stimuli  presented  are 
meaningful  (Glaser  &  Chi,  1988;  Anzai,  1991).  However,  this  apparent  advantage  disappears 
when  randomly  arranged  versions  of  the  same  stimuli  are  presented  (Ericsson  &  Hastie,  1994). 
Findings  of  this  nature  have  been  replicated  in  domains  as  diverse  as  chess,  bridge,  music, 
electronics,  and  computer  programming. 

In  sum,  the  expert’s  advantage  rests  on  being  able  to  quickly  attain  an  integrated 
representation  of  the  problem  situation  based  on  deep  rather  than  superficial  features  (Anderson, 
1993).  Accessing  relevant  schemata  makes  the  problem  meaningful  and  shifts  the  processing 
burden  from  working  memory  to  long  term  memory. 

Core  Processes 

Problem  representation,  discussed  above,  is  the  first  and  possibly  most  crucial  step  in 
most  accounts  of  the  core  processes  that  underlie  thinking  and  problem-solving.  When  people 
solve  problems,  not  only  is  knowledge  -  an  organized  set  of  facts  and  principles  related  to  some 
domain  (i.e.,  schemata)  -  important,  but  so  too  are  the  cognitive  processes  that  contribute  to  the 
effective  application  of  extant  knowledge  structures  in  solution  generation.  Interestingly,  one 
finding  from  expert-novice  research  is  that  domain-specific  knowledge  does  not  differentiate 
consistently  experts  from  novices  when  solving  novel  problems  (Centner  &  Block,  1983; 
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Mumford  &  Connelly,  1991).  Therefore,  in  addition  to  knowledge,  more  general  production 
processes  are  necessary  in  order  to  successfully  solve  problems. 

Several  researchers  have  proposed  stepwise  conceptions  of  problem-solving  (Nickerson, 
1994),  which  seek  to  identify  and  prescribe  the  sequence  of  steps  (i.e.,  the  process  of  thoughts 
and  actions)  people  take  to  solve  a  problem.  Like  decision  researchers,  problem-solving 
researchers  have  often  been  concerned  with  developing  prescriptions  for  improving 
decision/problem-solving  processes  in  addition  to  describing  basic  mechanisms.  Three  popular 
prescriptive  conceptions  include  those  proposed  by  Polya  (1957),  Hayes  (1981),  and  Bransford 
and  Stein  (1984).  Most  models  are  variations  on  the  first  proposed  by  Polya  (1957)  who 
suggested  that  four  steps  comprise  problem-solving;  (a)  understanding  the  problem,  (b)  devising 
a  plan,  (c)  carrying  out  the  plan,  and  (d)  looking  back.  Building  on  these  basic  steps,  Hayes 
proposed  a  six-step  model:  (a)  finding  the  problem,  (b)  representing  the  problem,  (c)  planning 
the  solution,  (d)  carrying  out  the  plan,  (e)  evaluating  the  solution,  and  (f)  consolidating  gains. 
Lastly,  Bransford  and  Stein  (1984)  proposed  five  “IDEAL”  stages  of  problem-solving:  (a) 
Identify  the  problem,  (b)  Define  and  represent  the  problem,  (c)  Explore  possible  strategies,  (d) 
Act  on  the  strategies,  and  (e)  Eook  back  and  evaluate  the  effects  of  your  activities. 

Eew  studies  have  actually  tested  the  descriptive  validity  or  prescriptive  utility  of  these 
kinds  of  stepwise  models  of  problem-solving  (Eipshitz,  Eevy,  &  Orchen,  2006).  Nevertheless, 
common  across  these  and  other  similar  models  of  the  problem-solving  process  are  certain 
fundamental  steps  (Eubart,  2001;  Reiter-Palmon  &  lilies,  2004);  (a)  defining  a  problem-solving 
goal,  (b)  accessing  relevant  information,  (c)  building  a  solution  from  this  information,  and  (d) 
evaluating  and  refining  the  proposed  solution.  Table  4  summarizes  each  of  the  aforementioned 
models  and  shows  for  which  stages  or  steps  the  models  are  aligned  similarly  or  not. 

An  important  implication  of  laying  out  the  process  in  steps  is  the  notion  that  problem¬ 
solving  is  fundamentally  linear  in  nature.  There  is  evidence  to  suggest,  however,  that  when 
engaged  in  solving  real  world  problems,  people  do  not  mentally  progress  through  an  orderly  and 
predictable  sequence  of  steps.  This  mirrors  the  departure  in  judgment  and  decision-making 
research  from  normative  models  to  those  highlighting  the  role  of  heuristics  and  biases.  As  Nutt 
(1984)  concluded,  the  linear  process  that  stepwise  or  phase  models  describe  “. .  .seems  rooted  in 
rational  arguments,  not  behaviour”  (p.  446). 

Eor  example,  early  scholars  of  managerial  problem-solving  assumed  that  managers  solve 
problems  in  a  rational,  linear  fashion  (Wagner,  1991).  While  a  rational  approach  is  potentially 
advantageous  for  reasons  related  to  training,  research  findings  do  not  suggest  that  leaders  adhere 
to  a  strict  and  consistently  applied  sequence  of  processes  in  dealing  with  complex,  ambiguous 
workplace  problems.  Instead,  it  seems  the  application  of  different  problem-solving  processes  is 
much  more  recursive.  Similarly,  Mintzberg,  Raisinghani,  and  Theoret  (1976)  found  that  rather 
than  follow  a  step-by-step  sequence  from  problem  definition  to  problem  solution,  managers 
typically  searched  the  problem  space  with  only  a  vague  notion  about  the  problems  with  which 
they  were  dealing.  Eurthermore,  managers  reported  having  little  idea  about  what  the  ultimate 
solution  would  be  until  they  found  it. 
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Table  4 


Summary  of  Cognitive  Processes  Associated  with  Problem-Solving 


Scholar(s) 

Cognitive  Processes  transform  the  initial  state  of  a  problem  into  a  goal  state 

Reiter-Palmon  & 
lilies  (2004) 

Defining  a  problem-  Accessing  relevant 

solving  goal  information 

Building  a  solution  from  this  information 

Evaluating  and  refining 
the  proposed  solution 

Polya  (1957) 

Understanding  the  problem 

Devising  a  plan 

Carrying 
out  the  plan 

Looking  back 

Hayes  (1981) 

Finding  the  problem 

Representing  the 
problem 

Planning  the  solution 

Carrying 
out  the  plan 

Evaluating 
the  solution 

Consoli 

dating 

gains 

Bransford  and 
Stein  (1984) 

Identify  the 
problem 

Define  and  represent  the 
problem 

Explore  possible  strategies 

Act  on  the 
strategies 

Look  back  and  evaluate 
the  effects  of  your 
activities 

Mumford, 

Mobley, 

Uhlman,  Reiter- 
Palmon,  & 

Doares  (1991) 

Problem 

construction 

Information 

encoding 

Category 

search 

Specification  of 

best-fitting 

categories 

Category 
combination  & 
reorganization 

Idea 

evaluation 

Solution 

implementa 

tion 

Solution  monitoring 

Davidson, 

Denser,  & 
Sternberg  (1994) 

Identifying 
and  defining 
the  problem 

Mentally  representing  the  problem 

o  Selective  encoding 

o  Selective  combination 

o  Selective  comparison 

Planning  how  to  proceed 

Solution  evaluation 
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In  a  related  study,  Isenberg  (1986)  eompared  the  verbal  protoeols  of  12  general  managers 
and  three  eollege  students  as  they  solved  a  short  business  ease.  Compared  to  students,  Isenberg 
found  that  experieneed  managers  began  planning  aetion  sooner,  asked  for  less  additional 
information,  made  more  inferenees  from  the  data,  and  were  less  refleetive  about  what  they  were 
doing  and  why.  Furthermore,  the  best  predietor  of  rated  effeetiveness  of  proposed  problem 
solutions  was  the  degree  to  whieh  the  managers  reasoned  analogieally  from  their  personal 
experienee.  The  key  departures  from  the  rational  model  involved  managers  engaging  in  aetion 
early  in  the  problem-solving  proeess,  eondueting  eursory  versus  exhaustive  analyses,  and  basing 
analyses  on  personal  experienee  with  analogous  problems  instead  of  using  more  formal 
prineiples  of  problem-solving.  Aeeounts  sueh  as  these,  whieh  portray  naturalistie  problem¬ 
solving,  further  underseore  the  relevanee  of  approaehes  like  the  AT  and  its  fast  and  frugal 
heuristie  eontent  for  helping  us  understand  MJP. 

In  the  problem-solving  literature,  the  applieation  of  heuristies  elearly  distinguishes 
experts  from  noviees.  Researeh  shows,  for  example,  that  experts  make  more  frequent  use  of 
heuristies  sueh  as  making  qualitative  representations  of  the  problem,  eonsidering  analogous 
problems,  and  analyzing  problems  into  manageable  eomponents.  In  eomparison,  noviees  are 
mueh  less  likely  to  do  so  (Niekerson,  1994).  In  large  part  this  is  beeause  their  representation  of 
the  problem  is  less  well  developed,  along  with  their  network  of  relevant  sehemata.  Glaser  (1996) 
notes  that  in  solving  problems  noviees  tend  to  display  individual  aets  and  judgments,  whereas 
experts  are  able  to  integrate  the  latter  into  overall  strategies.  Similarly,  pereeptual  learning  starts 
with  a  foeus  on  isolated  variables,  but  over  time  and  with  experienee,  this  shifts  to  pereeption  of 
eomplex  patterns.  Lastly,  Glaser  highlights  the  faet  that  experts  demonstrate  an  inereased  self- 
relianee  and  ability  to  form  new  strategies  as  neeessary. 

Not  only  do  experts  make  more  judieious  use  of  eertain  strategies,  but  evidenee  suggests 
they  also  rely  heavily  on  heuristies  that  are  relatively  speeifie  to  a  domain.  An  expert  physieist, 
for  example,  is  able  to  represent  problems  and  apply  problem-solving  strategies  in  ways  that  a 
noviee  eannot.  Sueh  strategies  are  highly  speeifie  to  the  domain  of  physios  problems  and  oannot 
easily  be  applied  more  broadly.  In  oontrast,  domain  independent  strategies  (e.g.,  means-ends 
analysis)  oan  be  used  effeotively  aoross  most  domains,  but  are  not  as  powerful  if  applied  to  a 
speeifie  physios  problem.  Indeed,  there  is  an  important  trade-off  between  the  range  of 
applioability  of  a  heuristie  and  its  power:  The  more  widely  applioable  a  heuristie  is  (i.e.,  less 
domain  dependent),  the  less  powerful  it  will  be  in  any  partioular  situation  (Niekerson,  1994). 

In  researeh  examining  the  military  deoision-making  prooesses  of  experts  versus  noviees, 
military  experts  fooused  on  aspeots  of  the  situation  that  resembled  previous  situations  they  faoed, 
and  used  general  heuristies  to  ohoose  a  satisfaotory  oourse  of  aetion  (Morrison,  Kelly,  Moore, 
Hutohins,  Cannon-Bowers,  &  Salas,  1998).  However,  more  experieneed  offieers  are  more  likely 
to  reeognize  that  any  given  situation  might  not  fit  a  previous  situation,  and  are  more  likely  to 
reeognize  the  situation  as  being  more  ambiguous  than  less  experieneed  offieers  (Cohen, 

Freeman,  &  Thompson,  1998).  In  taetieal  situations,  military  experts  are  more  likely  to  foeus  on 
eharaeteristies  of  a  partioular  enemy  for  that  speeifie  situation,  while  noviee  deoision  makers 
may  only  foeus  on  the  range  of  the  enemy’s  weapons  (Cohen  et  ah,  1998).  In  this  manner, 
experts  are  able  to  better  judge  the  situation  and  buy  more  time  to  make  the  deoision  (Cohen  et 
ah,  1998).  Another  key  differenoe  between  experts  and  noviees  in  the  military  is  an  expert’s 
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consideration  of  the  seeond  and  third  order  effeets  of  their  deeisions  and  the  larger  implications 
of  their  course  of  aetion.  Noviees  are  more  likely  to  focus  on  the  immediate  outcome  of  their 
deeision,  such  as  survival  of  their  unit  and  the  neutralization  of  the  enemy  (Cohen  et  ah,  1998). 

To  summarize,  in  highly  novel  situations,  the  effective  applieation  of  varied  problem¬ 
solving  proeesses,  in  addition  to  any  relevant  knowledge,  is  advantageous  to  experts.  Rather  than 
apply  these  proeesses  in  a  linear  manner,  however,  it  seems  that  experts  are  more  recursive. 
Moreover,  experts  rely  on  a  repertoire  of  domain-specifie  heuristies  and  other  problem-solving 
strategies. 

Problem  Type 

For  the  purposes  of  the  current  projeet,  we  are  foeused  on  novel,  ambiguous  problem 
situations  that  allow  for  discretion  on  the  part  of  the  Soldier,  and  whieh  arise  out  of  complex  as 
well  as  dynamic  tactical,  cultural,  and  legal/ethical  situations  and  environments.  In  these  real 
world  settings,  in  which  time  is  short  and  demands  are  many.  Soldiers  are  required  to  generate 
solutions  to  complex,  rapidly  unfolding  problems.  Moreover,  these  problems  are  embedded 
within  a  host  of  restrictions  imposed  by  constraints  on  time  and  resources,  by  system  demands, 
and  by  eonflieting  goals.  We  turn  now  to  exploring  expertise  specifically  within  the  context  of 
these  problem  settings. 

An  important  distinction  exists  between  well-defined  problems  that  have  a  known  goal,  a 
known  methodology,  a  way  to  reach  an  answer,  and  one  eorrect  answer  (which  may  not  be 
known  to  the  problem-solver,  but  is  known  to  others),  and  ill-defined  problems  (see  Sternberg, 
2000)  that  have  multiple  possible  goals,  multiple  ways  of  being  solved,  and  multiple  possible  and 
acceptable  solutions  (Reiter-Palmon  &  lilies,  2004).  This  latter  type  of  problem  is  most 
representative  of  the  kind  of  situation  on  which  the  eurrent  project  is  focused.  When  faced  with 
ill-defined  problems,  military  leaders  must  first  define  the  problem  or  situation  at  hand.  This 
involves  problem  representation,  whieh  as  we  have  seen  is  the  key  eognitive  proeess  that 
provides  the  context  for  the  application  of  the  other  processes.  Research  shows  that  problem 
construetion  occurs  relatively  automatically,  with  problems  being  framed  in  ways  that  fit  with 
the  problem-solver’s  past  experiences  and  sehemata  (Reiter-Palmon  &  lilies,  2004). 

One  attribute  of  ill-defined  problems  is  that  they  are  generally  solved  in  the  context  of 
ambiguous  information.  Marshall-Mies,  Fleishman,  Martin,  Zaccaro,  Baughman,  &  MeGee 
(2000)  suggest  that  in  eomplex  and  dynamic  environments,  leaders  must  solve  problems  under 
conditions  of  considerable  uncertainty.  It  is  seldom  fully  clear  what  information  is  either  relevant 
or  irrelevant.  Similarly,  it  may  be  difficult  to  obtain  accurate,  timely  information  and  identify  the 
key  diagnostic  information.  This  makes  information  search  and  encoding  another  key  cognitive 
proeess  essential  to  suecessful  problem-solving.  Information  can  be  gathered  from  both  internal 
sources  (e.g.,  schemata,  experience)  and  external  sources  (e.g.  others’  memories  and  experiences, 
the  internet).  In  either  case,  the  results  of  that  search  need  to  be  eonneeted,  synthesized,  and 
eneoded  so  as  to  facilitate  solution  produetion  (Reiter-Palmon  &  lilies,  2004). 


19 


When  people  eneounter  a  problem,  the  schemata  most  related  to  that  problem  are 
activated,  and  concepts  connected  to  those  schemata  are  retrieved.  Searching  and  encoding 
activities  are  principally  guided  by  cues  resulting  from  the  problem  representation  process 
(Reiter-Palmon  &  lilies,  2004).  These  cues  activate  relevant  schemata,  facilitating  the  retrieval  of 
concepts  connected  to  them.  But,  the  advantages  of  having  many  search  cues  will  only  be 
realized  if  there  is  sufficient  knowledge  for  these  additional  cues  to  activate.  Therefore,  in 
addition  to  its  role  in  problem  representation,  domain-specific  knowledge  is  also  important  for 
guiding  information-search  and  for  structuring  the  encoding  of  new  information,  especially  for 
ill-defined  problems.  Reiter-Palmon  and  lilies  (2004)  indicate  that  because  the  information  most 
related  to  a  problem  is  activated  automatically  and  quickly,  it  will  have  the  most  influence  over 
the  structure  that  generates  solutions.  This  is  akin  to  the  satisficing  heuristic  discussed 
previously,  in  which  time  pressures  in  naturalistic  settings  dictate  that  a  situation  or  problem  be 
resolved  quickly  and  effectively,  but  not  necessarily  perfectly.  This  is  unless  a  concerted  effort  is 
made  to  undertake  additional  search  and  encoding  activities.  Therefore,  in  response  to  problems 
about  which  information  is  ambiguous,  it  is  helpful  (time  permitting)  to  expend  further  cognitive 
effort  and  expand  search  and  encoding  of  information  beyond  that  initially  activated.  This 
description  has  clear  parallels  to  Kahneman  and  Frederick’s  (2005)  description  of  how  deliberate 
processes  inherent  in  System  2  are  accessed  in  an  effort  to  determine  the  quality  of  the  intuitive 
judgment  produced  by  System  1 .  While  effective  under  certain  circumstances,  many  field 
settings  may  not  permit  a  more  deliberate  evaluation  of  the  problem  and  possible  solutions. 

Another  attribute  of  ill-defined  problems  is  that  they  tend  to  be  novel.  Rather  than 
allowing  for  the  application  of  extant  schemata  in  a  rote  fashion,  novel  problems  present 
unfamiliar  parameters  and  require  atypical  and  adaptive  solutions.  Reiter-Palmon  and  lilies 
(2004)  suggest  that  generating  novel  solutions  necessitates  that  individuals  search,  combine,  and 
encode  concepts  from  categories  that  do  not  on  the  surface  appear  to  be  related.  This 
combination  and  reorganization  of  extant  schemata  generates  the  new  understandings  that  lay  the 
basis  for  novel  solutions  (Mumford  &  Connelly,  1991).  The  appropriate  use  of  heuristic 
strategies  is  also  key  to  novel  solution  generation.  Further,  problem-solvers  must  be  able  to 
evaluate  ideas  and  identify  one  or  a  set  that  work  the  best  for  the  current  situation.  This 
additionally  requires  identifying  potential  pitfalls  and  difficulties  that  may  arise.  While  idea 
generation  is  about  divergent  production,  idea  evaluation  is  more  about  convergent  thinking  to 
figure  out  which  solutions  should  ultimately  be  implemented.  In  doing  so,  problem-solvers  need 
to  be  able  to  evaluate  a  chosen  solution  against  competing  goals  and  identify  both  short-term  and 
long-term  consequences  of  implementing  a  particular  course  of  action.  Reiter-Palmon  and  lilies 
(2004)  note  that  identifying  as  many  consequences  and  obstacles  as  possible  means  that  the 
problem-solver  can  formulate  a  better  implementation  approach  as  well  as  develop  contingency 
plans  in  case  a  solution  is  ineffective  or  cannot  be  implemented  successfully. 

In  sum,  the  way  a  problem  is  represented  influences  cue  detection,  information  search, 
and  solution  generation.  Therefore,  problem  representation  is  especially  important  in  handling 
ill-defined  problems,  which  tend  to  be  ambiguous,  novel,  and  include  parameters  such  as  time 
pressure,  few  resources,  and  limited  information.  Expertise  in  handling  ill-defined  problems  also 
involves  searching  and  encoding  activities  that  are  guided  by  cues  resulting  from  the  problem 
representation  process.  The  use  of  heuristics  helps  to  explain  why  knowledge  and  information 
most  related  to  a  problem  is  activated  automatically  and  quickly  and  has  the  most  influence  in 
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solution  generation.  Expert  problem-solvers  in  poorly  defined  situations  are  able  to  evaluate 
ideas  and  identify  the  one  or  set  that  works  best  for  the  eurrent  situation.  This  requires  also 
identifying  potential  pitfalls  and  diffieulties  that  may  arise. 

Hunt  (1994)  suggests  that  a  pieture  emerges  of  an  effeetive  problem-solver  as  being  one 
who  has  memorized  a  large  number  of  trieks  of  the  trade,  so  to  speak,  rather  than  one  who  has 
developed  a  single,  powerful  strategy  for  reasoning.  He  argues  that  researeh  findings  point  to 
“. .  .loeal  optimality  aequired  at  the  expense  of  global  eonsisteney”  (p.  394).  In  other  words, 
people  learn  what  to  do  in  loeal  situations  rather  than  learning  abstract  principles.  Medin  and 
Ross  (1989)  propose  that  reasoning  is  case-based,  and  results  from  the  accumulation  and 
organization  of  domain-specific  examples  rather  than  the  generation  of  abstract  principles. 
Training  Soldiers  on  domain  specific  principles  within  situational  vignettes,  however,  has  been 
demonstrated  to  be  effective  in  training  military  decision-making  (Shadrick,  Crabb,  fussier,  & 
Burke,  2007).  Nevertheless,  interesting  parallels  can  be  drawn  to  Gigerenzer  and  colleagues’ 
notion  of  the  AT  in  which  fast  and  frugal  heuristics  take  advantage  of  environmental  constraints. 
The  NDM  approach,  discussed  next,  similarly  focuses  on  the  constraints  inherent  in  field  settings 
because  they  “. .  .establish  the  key  eliciting  conditions  for  making  decisions  and  shape  decisions 
through  their  constraints  and  affordances”  (Lipshitz  et  ah,  2001,  p.  334).  Expertise  from  the 
NDM  perspective  is  equally  about  the  proficient  decision  maker  as  it  is  about  the  field  settings  in 
which  they  operate.  In  the  next  section  we  will  examine  both  in  greater  depth. 

Naturalistic  Decision-Making 

In  everyday  life,  expertise  and  decision-making  are  relatively  inseparable.  With  the 
exception  of  purely  psychomotor  areas,  it  is  expected  that  identified  experts  make  good 
judgments  and  decisions  in  their  domain  of  expertise  (Herbig  &  Glockner,  2009).  It  is 
informative  to  examine  judgment  and  decision-making  at  different  levels  of  expertise,  which  is 
typically  signified  by  individuals  who  have  achieved  exceptional  skill  in  a  particular  domain. 
Indeed,  as  was  discussed  above,  a  key  distinction  that  separates  experts  from  novices  appears  to 
be  the  breadth  and  depth  of  their  domain-specific  knowledge  (Phillips,  Klein,  &  Sieck,  2004). 
While  a  great  deal  of  research  studying  expertise  has  been  conducted  in  laboratory  settings, 
approximating  expert  performance  under  controlled  conditions  is  somewhat  limited.  The 
naturalistic  approach  to  conducting  research  on  expert  judgment  is  exemplified  by  the  NDM 
framework  (see  Klein,  Orasanu,  Calderwood,  &  Zsambok,  1993;  Eipshitz,  Klein,  Orasanu,  & 
Salas,  2001;  Salas  &  Klein,  2001;  and  Zsambok  &  Klein,  1997  for  a  review). 

The  NDM  is  the  study  of  expert  judgment  and  decision-making  and  began  in  the  1990s  as 
a  line  of  research  juxtaposed  with  traditional  paradigms  of  judgment  and  decision-making 
research.  The  theoretical  foundation  of  NDM  researchers  is  in  basic  agreement  with  the  HB 
research  program,  in  that  both  perspectives  concur  that  human  decision-making  is  qualitatively 
different  from  normative  theories,  and  that  this  is  true  of  both  experts  and  novices  (Phillips,  et 
ah,  2004).  Yet,  NDM  is  related  to  ecological  rationality  because  it  focuses  on  simple  decision¬ 
making  heuristics  that  are  task  specific  (versus  general  purpose)  and  computationally  fast  and 
frugal  (versus  optimizing). 
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The  NDM  research  has  primarily  examined  how  experts  make  decisions  in  environments 
characterized  by  time  pressure,  ill-defined  goals,  ambiguity,  and  high  value  outcomes  of 
decisions  (Ross,  Shafer,  &  Klein,  2006).  As  such,  the  subject  of  interest  is  real-world  decision 
makers  faced  with  meaningful  decisions  about  which  they  have  relevant  knowledge  or  expertise. 
While  the  emphasis  is  particularly  on  individuals  in  crisis  situations  (e.g.,  fire  fighters,  pilots, 
military  leaders),  Phillips  et  ah,  (2004)  suggest  that  findings  about  expert  judgments  and  decision 
processes  observed  from  these  domains  also  generalize  to  those  that  are  less  crisis-like. 

In  their  review  of  the  NDM  literature,  Lipshitz  and  colleagues  (2001)  state  that  one  of  the 
defining  characteristics  of  NDM  research  is  that  it  places  “. .  .the  human  (and  hence  boundedly 
rational)  proficient  decision  maker  at  its  center  of  interest  and  as  its  basis  for  prescription”  (p. 
333).  Beyond  placing  the  expert  decision  maker  as  its  focus,  NDM  theories  attempt  to  describe 
the  process  through  which  these  experts  make  a  decision.  Other  defining  characteristics  of  NDM 
theories,  as  described  by  Lipshitz  et  ah,  (2001),  include:  (a)  the  use  of  matching  techniques  by 
experts  in  the  decision-making  process  (for  a  description  of  matching  see  Lipshitz,  1994),  (b)  the 
application  of  NDM  to  domain-  and  context-specific  situations,  and  (c)  the  development  of 
empirically-based  descriptive  models  of  expert  decision-making. 

Research  on  naturalistic  decisions  has  uncovered  several  new  views  of  decision-making 
(Lipshitz  et  ah,  2001).  First,  expert  decision  makers  are  not  superior  to  novices  because  their 
mental  processing  begins  to  align  more  closely  with  that  of  normative  theories.  Rather,  expertise 
is  believed  to  lead  to  a  broader  and  more  refined  set  of  heuristic  processes  that  promote 
outstanding  performance  in  the  specific  domains  out  of  which  they  developed  (Phillips  et  ah, 
2004).  Here  we  see  parallels  with  the  AT  approach  proposed  by  Gigerenzer  and  colleagues. 
Beyond  the  use  of  fast  and  frugal  heuristics,  situation  assessment  is  also  a  crucial  component  of 
the  process  for  NDM  types  of  decisions.  Situation  assessment  involves  the  sizing  up  and 
construction  of  a  mental  picture  of  a  situation  such  that  the  decision  maker  identifies  and  defines 
the  problem  at  hand  (Lipshitz,  1993).  This  is  much  like  the  problem  representation  process 
discussed  in  conjunction  with  problem-solving.  Following  situation  assessment,  the  next  most 
important  process  is  option  generation.  However,  Klein’s  (1998)  focal  conclusion  is  that, 
contrary  to  the  traditional  definition  of  decision-making  as  choosing  among  alternatives, 
proficient  decision  makers  rarely  compare  among  alternatives.  Instead,  they  assess  the  nature  of 
the  situation  and,  based  on  this  assessment,  select  an  action  appropriate  to  it.  This  set  of 
observations  led  Klein  (1998)  to  propose  what  he  calls  the  Recognition-Primed  Decision  (RPD) 
model. 

Recognition-Primed  Decision  (RPD)  Model 

In  novel,  ambiguous,  rapidly  changing  situations,  there  is  not  enough  time  to  apply 
normative  choice  rules.  Instead,  experienced  decision  makers  follow  a  recognition-primed 
decision-making  strategy,  which  involves  identifying  a  single  course  of  action  through  pattern 
matching.  The  Recognition-Primed  Decision  (RPD)  model  is  the  prototypical  NDM  model 
(Lipshitz  et  ah,  2001).  It  describes  how  experts  rely  on  extensive  knowledge  to  make  judgments 
about  situations  and  decide  how  to  act  (Phillips  et  ah,  2004).  It  was  developed  based  on 
extensive  interviews  with  experts  in  varied  domains,  ranging  from  firefighters  to  military 
command  and  control. 
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It  is  important  to  note  that  domain  specific  knowledge  and  experience  play  a  crucial  role 
in  the  RPD  model.  RPD  cannot  be  executed  without  such  knowledge.  As  was  mentioned  in  the 
discussion  on  problem-solving,  experience  and  knowledge  are  essential  to  assigning  meaning  to 
information^  Phillips  et  ah,  (2004)  describe  three  variations  of  the  model.  The  first  pertains  to 
familiar  situations  that  are  judged  to  be  routine.  In  such  cases,  decisions  are  thought  to  be 
intuitive  and  automatic.  Proficient  decision  makers  understand  situations  by  recognizing  them  as 
prototypical  situations  they  have  experienced  previously  and  stored  in  long-term  memory.  A 
recognition  match  occurs,  without  deliberation,  and  becomes  the  basis  for  a  course  of  action  or 
decision. 

The  second  variation  exists  to  explain  decision-making  in  ambiguous  or  unfamiliar 
situations.  In  such  cases,  experts  think  about  the  situation  at  hand  and  often  search  for  more 
information  to  augment  their  preliminary  assessment.  Sometimes,  this  step  is  conducted  based  on 
the  need  to  reconcile  an  inconsistency  or  unexpected  piece  of  information.  In  either  case,  in  the 
second  variation  of  the  RPD  model,  proficient  decision  makers  put  forth  effort  to  generate  an 
accurate  assessment  of  the  situation.  In  doing  so  a  clear  course  of  action  becomes  apparent. 
Particularly  for  this  variation,  mental  simulation  can  be  used  to  play  out  a  story  about  the 
precursors  to  the  current  situation.  The  story  that  is  most  plausible  further  helps  the  decision 
maker  to  understand  what  is  going  on  and  arrive  at  a  reasonable  course  of  action. 

In  the  third  and  final  variation  of  the  RPD  model,  prior  to  engaging  in  action,  decision 
makers  take  steps  via  mental  simulation  to  evaluate  the  initial  course  of  action  selected  by  the 
recognition  match.  In  other  words,  the  effectiveness  of  the  selected  course  of  action  is  tested 
mentally.  This  process  continues  until  a  course  of  action  is  settled  upon  that  passes  the  test  of 
mental  simulation. 

The  RPD  model  is  functional  in  that  it  does  not  seek  to  describe  how  decisions  should  be 
made,  but  rather  how  they  are  actually  made  by  experts  in  specific  domains  (Lipshitz,  1993). 
What  the  model  proposes  is  that  experts  focus  on  critical  cues  and  identity  causal  factors. 
Focusing  on  these  elements  lessens  information  overload  and  confusion,  both  of  which  can 
paralyze  novice  decision  makers  in  stressful,  rapidly  changing  situations.  The  identification  of 
cues  and  causal  factors  helps  to  establish  accurate  expectations,  which  together  with  plausible 
goals,  are  key  to  selecting  an  appropriate  course  of  action.  Further,  serial  selection  on  the  basis  of 
typicality  means  that  reasonable  actions  can  be  selected  quickly.  Lastly,  mental  simulation 
promotes  critical  thinking  (Lipshitz,  1993).  Like  experts  studied  in  the  problem-solving 
literature,  proficient  decision  makers  make  use  of  the  satisficing  heuristic.  In  naturalistic  settings, 
time  pressure  often  necessitates  quick  solutions.  A  perfect  solution  is  rarely  deemed  necessary; 
instead,  the  solution  simply  has  to  be  effective  in  the  context  of  the  situation  at  hand. 


*  Funder  (1987)  noted  that  good  judgment  is  the  ability  to  go  beyond  the  information  given  and  to  rely  on  one’s 
broader  knowledge  and  past  experienee.  Authorities  on  artificial  intelligence  (AI)  agree  that  a  key  difficulty  in 
getting  computers  to  understand  language  or  solve  natural  problems  is  that  computers  use  only  the  information  they 
have  been  explicitly  given,  rather  than  the  broad  array  of  real-world  knowledge  and  experience  upon  which  humans 
rely.  While  psychologists  might  seek  to  eliminate  human  cognitive  biases  from  the  judgment  process,  such  biases 
are,  according  to  Funder  (1987),  the  very  same  things  that  researchers  in  AI  are  finding  necessary — but  extremely 
difficult — to  incorporate  into  that  process. 
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The  NDM  researchers  go  beyond  the  HB  approach  in  terms  of  the  role  of  heuristics. 
While  both  approaches  agree  that  human  judgment  and  decision-making  is  qualitatively  different 
from  normative  theories,  the  RPD  program  goes  further  than  HB  by  dealing  with  issues  of 
representation  and  process  (Phillips  et  ah,  2004).  Depending  on  how  a  problem  situation  is 
represented,  a  range  of  different  heuristics  can  be  brought  to  bear.  In  this  sense,  NDM  is  much 
more  aligned  with  the  basic  premises  of  the  AT  model,  and  ecological  rationality  in  particular.  In 
much  the  same  way  as  fast  and  frugal  heuristics  are  conceived  of  taking  advantage  of  structures 
in  the  environment,  so  too  does  NDM  regard  heuristics  as  inextricably  related  to  the  task  or 
knowledge  domain  to  which  they  are  applied.  In  order  to  effectively  apply  a  heuristic,  the 
problem  must  first  be  defined  in  some  fashion.  This  can  be  done  quickly  and  automatically,  or 
more  slowly  and  thoughtfully.  This  perspective  is  similar  in  many  respects  to  the  problem¬ 
solving  outlook  on  expertise,  which  highlights  problem  representation  as  crucial  to  solution 
generation.  According  to  NDM,  expert  decision  makers  are  not  better  than  novices  because  their 
processing  starts  to  look  more  like  that  of  normative  theories.  Rather,  expertise  leads  to  a  broader 
and  more  refined  set  of  heuristic  processes  that  promote  high  performance  within  the  specific 
task  domains  to  which  they  are  adjusted  (Phillips  et  ah,  2004). 

Klein  suggests  that  decision  makers  manage  to  act  proficiently  under  time  pressure  by 
relying  on  domain  specific  expertise,  which  enables  them  to  identify  the  situation  quickly  and 
accurately  and  to  act  promptly  by  accessing  an  action  queue  that  is  arranged  with  the  appropriate 
action  on  top  (Lipshitz,  1993).  Experts’  success  at  this  is  influenced  greatly  by  their  ability  to 
identify  important  cues  and  causal  factors,  out  of  which  primed  recognition  can  take  place. 

The  judgment  and  decision-making  of  experts  is  conceived  as  relatively  intuitive, 
although  the  RPD  model  allows  the  decision  maker  an  opportunity  to  deliberate  in  unfamiliar 
circumstances  in  order  to  assess  the  situation  accurately.  This  is  similar  to  the  ASM  model  in  that 
both  System  1  and  System  2  contribute  to  the  final  judgment,  which  starts  with  a  more  intuitive 
process  that  is  evaluated  by  a  more  rational  process.  Interestingly,  Hammond  (of  Social 
Judgment  Theory  and  Cognitive  Continuum  Theory)  recognized  that  human  intuition  is 
particularly  useful  because  it  can  allow  for  the  recognition  of  rare  but  diagnostic  cues  that  are 
hard  to  incorporate  into  purely  analytic  judgment  and  decision-making.  This  seems  to  be 
precisely  what  NDM  and  AT  researchers  wish  to  highlight  about  the  use  of  situationally-adjusted 
heuristics.  Klein,  more  in  keeping  with  ASM  model,  suggests  a  two  phase  sequence  in  which  a 
typically  quick  preliminary  selection  based  on  matching  is  followed  by  more  deliberate 
evaluation  (i.e.,  mental  simulation). 

In  all  three  cases  (i.e.,  ASM  model,  AT  model,  and  NDM),  with  increasing  time  pressure, 
decision  makers  make  more  use  of  intuitive  decision  processes,  which  require  less  time  and 
effort  to  act.  As  such,  decision  makers  demonstrate  adaptability  to  the  decision  task  at  hand  and 
adopt  strategies  that  do  not  tax  limited  information  processing  capacity.  The  approaches  differ  in 
how  they  conceive  of  the  role  of  the  environment  in  shaping  the  heuristics  used  and  whether  and 
to  what  degree  more  deliberate,  analytical  processes  are  accessed  to  refine  the  initial,  more 
intuitive  judgment.  In  the  next  section,  we  will  bring  together  the  various  themes  from  this 
literature  review  in  support  of  how  we  conceive  of  MJP,  linking  the  themes  from  the  literature 
with  the  four  key  components  of  MJP. 
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Military  Judgment  Proficiency:  A  Synopsis 


We  began  this  report  by  defining  military  judgment  proficiency  (MJP)  as  a  complex  skill 
that  involves  four  major  components:  (a)  accurately  framing  decision  problems,  (b)  detecting  and 
interpreting  important  situational  factors/cues,  (c)  forecasting  the  consequences  of  different 
courses  of  action,  and  (d)  choosing  an  effective  course  of  action  based  on  the  preceding  three 
factors.  The  research  and  theories  from  the  judgment  and  decision-making  and  problem-solving 
literatures  reviewed  here  suggest  that  each  component  -  individually  and  in  combination  -  is 
essential  to  demonstrating  good  judgment,  especially  under  novel,  ambiguous,  and  rapidly 
changing  circumstances.  In  this  section  we  will  summarize  the  key  links  between  the 
components  of  our  definition  on  MJP  and  the  extant  literature. 

Problem-Solving  and  Expert  Schemas 

At  the  outset  of  any  problem  situation,  an  important  skill  is  being  able  to  quickly  attain  an 
integrated  representation  and  accurately  frame  the  situation  in  one’s  mind.  Problem 
representation  is  a  key  cognitive  process,  one  of  several  highlighted  by  problem-solving 
researchers  as  contributing  to  the  effective  application  of  extant  knowledge  in  decision 
generation.  Proficient  judgment  requires  some  amount  of  domain-specific  knowledge  (Ericsson 
&  Hastie,  1994).  Knowledge  exists  in  structures  called  schemas  or  schemata,  which  themselves 
exist  in  networks  and  represent  an  individual’s  understanding  of  the  world.  Experts  have  large, 
rich,  well-organized  schemata  composed  of  declarative  knowledge  about  a  particular  domain. 
Problem-solving,  including  the  encoding  and  retrieval  of  information,  is  largely  schemata  driven 
(Eipshitz  et  ah,  2006).  It  is  important  to  note  that  experts  do  not  apply  different  or  better 
problem-solving  processes.  Rather,  their  superior  performance  can  be  attributed  to  the  nature, 
amount,  and  organization  of  knowledge  they  possess  about  a  particular  domain  (Chase  &  Simon, 
1973).  As  Hunt  (1994)  states,  experts  have  the  advantage  of  being  able  to  apply  previously 
learned  knowledge  as  well  as  rules  that  led  to  a  solution  in  the  past. 

The  Role  of  Heuristics 

The  reliance  of  experts  on  both  knowledge  and  rules  is  a  key  theme  in  this  literature 
review.  The  application  of  previously  learned  knowledge  in  conjunction  with  mental  rules  is  a 
hallmark  of  expertise.  Heuristics,  or  psychological  shortcuts,  are  particularly  well-adapted  rules 
for  helping  cognitively  constrained  humans  handle  the  challenges  imposed  by  various  situational 
constraints  on  time,  resources,  and  information.  Moreover,  they  represent  a  key  mechanism  that 
underlies  each  of  the  proposed  components  of  MJP.  Research  in  judgment  and  decision-making 
and  problem-solving  acknowledges  the  importance  of  heuristics  as  an  essential  human  adaptation 
to  the  task  environment.  Models  differ,  however,  in  whether  they  regard  such  heuristics  as 
advantageous  or  not.  Eor  example,  the  HB  approach  views  heuristics  independent  of  the 
environment  and  as  leaving  humans  susceptible  to  cognitive  biases,  which  results  in  suboptimal 
decisions.  In  contrast,  the  AT  and  NDM  approaches  take  the  view  that  heuristics  are  highly 
adapted  to  the  environment  out  of  which  they  developed,  and  are  therefore  of  substantial  benefit 
to  the  humans  that  apply  them. 
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A  Quasi-Rational  Perspective 

We  take  the  view,  in  aceordance  with  the  latter  approaehes  (i.e.,  AT  and  NDM),  that  in 
novel,  ambiguous,  and  rapidly  ehanging  situations  in  which  there  is  limited  time,  information, 
and  resources,  situational  framing,  assessment,  and  forecasting  happen  relatively  intuitively  - 
guided  by  an  underlying  set  of  domain-specific  schemata  and  heuristic  processes  that  promote 
high  performance  within  the  specific  task  domains  from  which  they  developed.  We  do  not, 
however,  conceive  of  MJP  as  an  entirely  intuitive  process.  We  suggest  that  in  many  situations  in 
which  the  timeframe  does  not  call  for  immediate  action,  a  somewhat  more  deliberate,  analytical 
approach  is  likely  to  be  taken. 

In  accordance  with  the  ASM  and  RPD  models,  we  argue  the  MJP  is  likely  to  involve 
some  analysis  that  serves  as  a  check  on  the  initial,  heuristically-driven  judgment.  This  means  it 
can  be  endorsed,  corrected,  or  overridden,  but  only  if  time  and  circumstances  permit.  This  quasi- 
rational  perspective  on  MJP  is  consistent  with  current  thinking  about  judgment  and  decision¬ 
making  in  naturalistic  settings.  The  RPD  model  is  particularly  useful  in  shedding  light  on  expert 
decision-making.  For  example,  it  highlights  a  two-phase  sequence  in  which  a  typically  quick 
preliminary  assessment  of  the  situation  (e.g.,  comparing  features  of  the  situation  to  prototypical 
situations  experienced  previously  and  stored  in  long-term  memory)  is  followed  by  more 
deliberate  evaluation  of  a  selected  course  of  action  (e.g.,  mental  simulation).  Moreover,  the 
forecasting  aspect  of  mental  simulation  in  the  RPD  model  is  particularly  intriguing  as  a  means  to 
evaluate  a  chosen  course  of  action  against  competing  goals  to  identify  both  short-term  and  long¬ 
term  consequences  of  implementing  a  particular  course  of  action. 

In  order  to  move  from  theory  to  measurement,  we  believe  it  is  most  fruitful  to  focus  here 
on  the  measurement  of  situational  framing,  assessment,  and  forecasting,  as  well  as  selecting  a 
course  of  action,  components  of  MJP  that  are  most  amenable  to  measurement  in  a  situational 
judgment  test.  In  the  past,  situational  judgment  tests  have  been  used  successfully  to  measure 
problem-solving  and  decision-making  skills  (e.g.,  Beal,  2007;  Waugh,  2004;  Motowidlo, 
Dunnette,  &  Carter,  1990).  While  the  distinction  between  intuitive  and  rational  judgment  is 
important,  and  a  pervasive  theme  throughout  this  literature  review,  it  is  not  a  distinction  that  can 
easily  be  measured  using  a  paper  and  pencil  situational  judgment  test.  Each  of  the  other 
components,  however,  is  highly  amenable  to  measurement  via  a  situational  judgment  test.  The 
kind  and  extent  of  knowledge  (in  the  form  of  schemata)  and  heuristics  that  underlie  these 
components  of  MJP  are  conceived  as  cognitive  mechanisms  that  can  only  be  derived  indirectly 
through  people’s  reports  of  how  they  frame  a  situation,  what  factors/cues  strike  them  as 
important,  and  what  outcomes  they  forecast  as  desirable.  In  the  next  section  we  discuss 
measurement  issues  at  greater  length  and  elaborate  on  methods  for  measuring  each  aspect  of 
MJP. 


Situational  Judgment  Test  (SJT)  as  an  Assessment  Tool 

In  organizational  and  educational  settings,  there  has  been  growing  interest  in  a  family  of 
measures  commonly  known  as  situational  judgment  tests  (SJTs)  (Motowidlo,  Dunnette,  & 
Carter,  1990).  Whereas  the  label  SJT  is  widely  used  in  employment  settings,  similar  instruments 
are  used  in  other  domains  under  the  label  “tacit  knowledge  tests”  or  “practical  judgment  tests” 
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(Sternberg,  Wagner,  Williams,  &  Horvath,  1995).  Regardless  of  speeifie  label,  these  instruments 
all  have  in  eommon  the  presentation  of  a  written  or  video-based  seenario  that  is  aeeompanied  by 
a  set  of  alternative  eourses  of  aetion,  from  which  the  individual  is  asked  to  make  choices.  A 
variety  of  different  response  options  are  possible  (McDaniel  &  Nguyen,  2001).  In  some  cases  the 
task  is  to  select  the  single  best  course  of  action,  or  the  worst  course  of  action.  Response  option 
formats  in  which  the  respondent  is  asked  to  judge  the  quality  of  different  options  are  known  as 
knowledge  type  formats.  Another  type  of  format  is  the  behavioral  tendency  format,  for  which  the 
respondent  is  asked  to  identify  the  action  they  would  most  or  least  likely  perform. 

There  are  a  number  of  reasons  why  SJTs  are  growing  in  popularity.  First,  asking 
examinees  to  respond  to  realistic  scenarios  is  appealing  because  it  seeks  information  about 
judgment  in  context.  This  is  in  contrast  to  the  de-contextualized  nature  of  many  standardized 
tests.  Second,  it  is  common  to  find  smaller  subgroup  differences  (e.g.,  race  or  gender  differences) 
for  measures  with  a  smaller  g-loading  than  with  traditional  cognitive  ability  tests  (Clevenger, 
Pereira,  Wiechmann,  Schmitt,  &  Harvey,  2001).  Thus,  the  addition  of  less  cognitively  loaded 
measures  holds  promise  as  a  means  of  simultaneously  increasing  validity  and  reducing  adverse 
impact.  Third,  large-scale  studies  have  shown  that  SJTs  have  substantial  criterion-related 
validities  (McDaniel,  Morgeson,  Finnegan,  Campion,  &  Braverman,  2001). 

Measuring  Specific  Constructs  Using  SJTs 

Several  attempts  have  been  made  to  measure  specific  constructs  using  SJTs.  For  instance, 
researchers  have  used  these  tests  to  measure  constructs  as  diverse  as  teamwork  knowledge  and 
skills  (Stevens  &  Campion,  1999),  interpersonal  skills  and  problem-solving  skills  (Motowidlo, 
Dunnette,  &  Carter,  1990),  adaptive  performance  (Arad,  Borman,  &  Pulakos,  1999),  integrity 
(Becker,  2004),  and  personality  (Motowidlo,  Hooper,  &  Jackson,  2006).  As  previously 
mentioned,  SJTs  have  been  used  successfully  to  measure  problem-solving  and  decision-making 
skills  (e.g.,  Beal,  2007;  Waugh,  2004;  Motowidlo,  Dunnette,  &  Carter,  1990).  In  addition  to 
using  SJTs  to  measure  specific  constructs,  these  tests  can  be  written  to  reflect  an  underlying 
model,  such  as  a  set  of  competencies,  attributes  derived  from  a  job  analysis,  review  of  the 
literature,  or  a  theory  of  effective  performance.  One  goal  of  this  effort  is  to  use  a  SJT  to  measure 
the  various  aspects  of  judgment  proficiency,  as  specified  by  theory  and  by  the  literature  review 
provided.  We  will  return  to  this  point  in  detail  below. 

Unfortunately,  most  efforts  to  develop  SJTs  that  measure  specific  constructs  have 
resulted  in  tests  that  are  multidimensional,  have  low  internal  consistency  reliability  estimates,  or 
otherwise  display  poor  psychometric  characteristics  (Weekley,  Ployhart,  &  Holtz,  2004).  Indeed, 
the  typical  SJT  correlates  with  cognitive  ability,  experience,  and  personality  (McDaniel  & 
Nguyen,  2001).  One  difficulty  in  designing  SJTs  to  measure  specific  constructs  is  that  their 
format  may  impose  constraints  on  the  type  of  constructs  measured.  For  instance,  because  SJTs 
by  their  very  nature  measure  judgment,  it  is  likely  not  possible  to  construct  an  SJT  that  does  not 
correlate  to  some  degree  with  cognitive  ability.  In  their  meta-analysis  of  the  criterion-related 
validities  of  SJTs,  McDaniel  et  ah,  (2001)  found  a  mean  observed  correlation  between  SJTs  and 
cognitive  ability  of  .36. 
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Chan  and  Schmitt  (2005)  argue  that  the  format  of  the  SJT  does  indeed  impose  eertain 
eonstraints  on  the  eonstruets  it  ean  measure.  They  argue  that  irrespeetive  of  the  eonstruet  a 
researeher  intends  to  measure,  the  dominant  eonstruets  measured  are  different  than  the  intended 
eonstruet.  Speeifieally,  they  propose  that  the  dominant  eonstruets  assessed  by  SJTs  -  regardless 
of  the  intended  eonstruet  to  be  measured  -  are  adaptability  eonstruets  that  are  likely  a  function  of 
individual  difference  traits  and  the  result  of  aequisition  through  previous  experienees,  and  a 
eontextual-knowledge  eonstruet  that  may  be  gained  through  experienee.  In  brief,  Chan  and 
Sehmitt  believe  the  dominant  eonstruet  assessed  by  all  SJTs  is  practieal  intelligenee,  eontextual 
knowledge,  or  simply  judgment  (Chan  &  Sehmitt,  2002,  Sternberg  et  ah,  2000).  Praetieal 
intelligenee  refers  to  the  ability  or  expertise  to  effeetively  respond  and  suoeessfully  adapt  to  a 
variety  of  praetieal  problems  or  situational  demands. 

Measuring  Judgment  Proficiency  Using  an  SJT 

In  the  eurrent  projeet,  we  are  developing  an  SJT -based  measure  of  MJP.  To  our 
knowledge,  sueh  a  measure  does  not  already  exist.  In  developing  the  current  SJT  we  sought  a 
somewhat  innovative  measurement  approach.  Unlike  typieal  SJTs,  whieh  present  a  seenario  and 
ask  respondents  to  provide  ratings  about  various  eourses  of  aetion  one  eould  take  in  the  eontext 
of  the  seenario,  we  developed  additional  judgment  question  types,  based  on  our  review  of  the 
literature  and  subsequent  definition  of  judgment  profieieney,  that  ask  respondents  to  provide 
ratings  that  demonstrate  how  they  evaluate  key  judgment  preeursors. 

Speeifieally,  we  believe  the  SJT  format  is  partieularly  amenable  to  measuring 
respondents’  ability  to  identify  the  most  eritieal  cues  and  situational  features  (e.g.,  situational 
awareness),  as  well  as  forecast  the  most  important  seeond-  and  third-order  effects  associated  with 
a  eourse  of  aetion.  In  this  manner,  we  are  not  only  taking  into  eonsideration  profieieney  in 
choosing  an  effective  eourse  of  aetion,  but  we  are  also  taking  into  eonsideration  two  major 
eomponents  of  that  judgment  and  deeision-making  proeess  in  naturalistie  settings:  (a)  Identifying 
the  most  relevant  aspeets  in  the  situation,  and  (b)  tying  the  eourse  of  action  to  a  desired  goal  or 
end-state.  As  was  highlighted  in  the  literature  review,  profieient  deeision  makers  (e.g.  experts) 
are  able  to  foeus  on  eritieal  cues  and  identify  causal  factors.  Based  on  this  rapid  assessment,  a 
course  of  aetion  is  seleeted  that  satisfies  the  requirement  of  the  situation.  In  our  SJT,  by  asking 
questions  matehing  these  three  eomponents,  we  are  not  only  able  to  look  at  eaeh  eomponent 
individually,  but  we  are  also  able  to  eombine  all  three  to  assess  optimal  solutions.  Beeause  SJTs 
are  eonsidered  low  fidelity  simulations,  it  is  necessary  to  show  how  the  results  relate  to  more 
realistie  experiences  sueh  as  field  training  exereises. 

SJT  Development  and  Measurement  Issues 

Based  on  the  assumption  that  an  SJT  is  an  appropriate  format  to  measure  judgment 
profieieney,  we  will  now  eonsider  several  additional  development  and  measurement  issues 
related  to  the  SJT  format.  These  inelude:  (a)  generation  of  item  eontent,  (b)  item  stem 
complexity,  (c)  response  option  generation,  (d)  response  instructions,  (e)  determining  the 
effectiveness  of  the  response  options,  and  (f)  seoring  the  SJT.  Eaeh  of  these  eonsiderations  will 
be  discussed  below,  paying  partieular  attention  to  how  eaeh  will  be  addressed  for  an  SJT -based 
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measure  of  judgment  profieiency.  We  will  use  an  example  item  throughout  this  seetion  to 
illustrate  various  aspects  of  the  SJT  under  development. 

Generating  SJT  Content 

A  preliminary  issue  is  how  to  determine  the  content  of  SJT  items.  The  critical  incident 
method  (Flanagan,  1954)  is  often  used  for  identifying  item  content.  Typically,  this  involves 
asking  subject-matter  experts  (SMEs)  to  recall  exceptionally  good  and  exceptionally  poor 
examples  of  performance.  The  SMEs  are  asked  to  name  the  antecedent,  or  what  led  to  the 
incident,  the  behavior  that  was  performed  that  was  exceptionally  good  or  poor,  and  the 
consequence(s)  of  the  behavior.  With  this  approach,  the  antecedent  will  often  become  the  stem 
for  the  SJT  item,  and  the  behavior  can  become  one  of  the  response  options. 

A  second  approach  is  to  write  SJT  items  to  conform  specifically  to  some  underlying  set 
of  competencies,  attributes  identified  in  a  job  analysis,  or  literature  review  detailing  attributes 
relevant  to  job  performance.  This  second  approach  aims  to  measure  specific  homogenous 
constructs.  There  are  advantages  to  both  methods.  Critical  incidents  provide  a  rich  source  of 
information,  but  can  be  expensive  and  time-consuming.  Construct-driven  approaches  help  to 
ensure  that  a  specific  construct  is  being  measured,  thereby  increasing  the  content  validity  of  the 
measure. 

Eor  the  SJT  under  development  here,  we  relied  primarily  on  an  approach  similar  to  that 
for  generating  critical  incidents.  Specifically,  we  conducted  focus  groups  with  SMEs,  who 
provided  rich  descriptions  of  situations  in  which  they  had  made  decisions  under  novel, 
ambiguous,  and  rapidly  changing  circumstances.  At  the  outset,  we  sought  examples  of  situations 
that  fall  into  each  of  the  four  categories  determined  at  the  start  of  the  project:  (a)  leader  tactical, 
(b)  cultural,  (c)  legal/ethical,  and  (d)  leader  non-tactical  situations.  Elsing  a  semi-structured 
interview  protocol,  researchers  gathered  information  about  the  background  of  the  situation, 
important  elements  (e.g.,  quantity  and  kind  of  resources,  amount  of  control,  time  pressure,  and 
other  constraints),  and  the  outcome. 

Interviews  were  conducted  with  nine  separate  groups  of  SMEs,  following  which  the 
research  team  drafted  a  set  of  sample  SJT  situations.  A  subsequent  set  of  SME  focus  groups 
provided  feedback  on  these  first  situations  as  well  as  additional  information,  based  on  which 
another  set  of  situations  were  drafted.  A  third  group  of  SMEs  gave  detailed  feedback  on  the 
complete  set  of  situations.  The  research  team  implemented  modifications  to  the  situations, 
accordingly.  An  example  situational  stem  from  the  tactical  scenarios  derived  from  these  focus 
groups  is  presented  below: 

“You  are  a  Team  Sergeant  located  in  a  small  Eorward  Operating  Base 
(EOB).  You  receive  credible  information  that  a  suicide  bomber  will  be  making  his 
way  towards  your  EOB  the  following  day.  You  know  what  particular  road  he  will 
take  and  that  his  intention  is  to  attack  the  EOB.  You  and  some  of  your  men  come 
up  with  a  plan  to  walk  to  a  position  15  km  from  the  EOB  to  set  up  an  ambush. 

You  plan  to  do  this  during  the  night  so  as  not  to  be  spotted  and  tip  off  to  the 
would-be  suicide  bomber.  Your  men  have  trained  the  local  host-nation  (HN) 
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military  and  police  force  about  how  to  conduet  night  patrols  and  security  checks 
in  your  lED-rich  area  of  operations.  Your  plan  involves  close  coordination  with 
the  local  police  and  security  forces  before  you  depart  so  as  to  avoid  eoming  in 
contact  with  them  while  you  are  doing  the  overnight  trek;  in  this  manner  avoiding 
potentially  catastrophic  friendly-on-friendly  fire.  You  have  been  training  these 
forces  for  about  6  months  and  you  know  that  they  will  be  eondueting  nightly 
patrols,  and  although  they  are  professional,  they  will  fire  upon  any  force  they 
encounter  at  night  before  asking  questions.  Just  before  leaving  the  base,  your  new 
team  leader,  a  Captain  with  1  week  of  experience  in  country,  tells  you  that  he 
does  not  agree  with  the  coordination  part  of  your  plan.  He  is  coneerned  that  the 
local  forces  might  be  eorrupt  and  tip-off  the  suieide  bomber  or  give  away  your 
position.  He  insists  that  you  do  the  hike  without  eoordinating  with  HN  forces.” 

The  SJT  provided  above  falls  into  the  leader  taetical  eategory.  It  should  be  noted  that  in 
contrast  to  the  other  situation  categories,  the  leader  non-tactical  situations  were  more  ehallenging 
to  define  clearly  and  seemed  to  eneompass  any  situation  that  was  not  already  included  in  the 
leader  tactieal,  cultural,  or  legal/ethical  categories.  As  sueh,  the  nature  of  the  situations  gathered 
initially  for  the  leader  non-tactieal  eategory  was  much  more  heterogeneous.  Further,  these 
situations  tended  not  to  conform  well  to  the  prineipal  criteria  we  established  for  deeiding  whether 
a  situation  could  be  used:  that  it  was  charaeterized  as  being  novel,  ambiguous,  and  rapidly 
changing.  As  such,  the  research  team  chose  to  eliminate  the  leader  non-tactical  category. 

Complexity  of  Stems  and  Stem  Fidelity 

A  second  issue  emerged  regarding  the  complexity  of  the  item  stems  and  whether  they 
should  be  relatively  simple  or  very  detailed.  Some  SJT  researehers  have  theorized  that  detailed 
stems  are  more  valid  beeause  they  better  represent  a  sample  of  performance  taken  direetly  from 
the  performance  domain  (McDaniel  et  ah,  2001).  In  keeping  with  this  hypothesis,  other 
researehers  have  argued  that  the  high  degree  of  correspondenee  between  the  predietor  and 
eriterion  domains  for  sueh  samples  of  performance  is  precisely  what  explains  the  high  eriterion- 
related  validities  of  SJTs  (Chan  &  Schmitt,  2002;  Lievens,  Buyse,  &  Sackett,  2005).  Finally, 
highly-detailed  and  speeific  stems  may  also  have  greater  face  validity  and  more  positive  test- 
taker  reactions. 

The  degree  of  detail  in  the  situation  ean  be  eritieal  because  even  small  changes  in  the 
amount  of  detail  have  been  shown  to  effect  how  the  situation  is  interpreted  (Brooks  & 

Highhouse,  2006).  Overall,  higher  detail  appears  to  be  preferable  to  less  detail  beeause  it  reduees 
the  opportunity  for  situation  eonstrual.  However,  care  must  be  taken  to  ensure  that  reading  level 
remains  appropriate  throughout.  Some  studies  (e.g.,  Saceo,  Seheu,  Ryan,  &  Schmitt,  2000; 

Sacco,  Schmidt,  &  Rogg,  2000)  have  shown  reading  level  to  be  related  to  subgroup  differenees. 
Thus,  if  sufficient  eare  is  not  taken,  detailed  stems  could  increase  the  ehanees  for 
misinterpretation. 

The  issue  of  level  of  detail  in  the  stem  is  related  to  the  issue  of  stem  fidelity.  Ideally,  item 
stems  should  have  high  psyehologieal  fidelity  in  the  sense  that  they  require  the  same  cognitive 
operations  as  tasks  performed  on  the  job.  In  the  case  of  assessing  judgment  proficieney,  a  high 
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level  of  detail  in  item  stems  is  appropriate  for  maintaining  high  psychologieal  fidelity.  This  is 
due  to  the  faet  that  the  situations  requiring  judgment  profieieney  in  dynamie,  ambiguous 
situations  require  appreeiation  of  how  to  weigh  many  situation-speeific  faetors  at  onee. 

Therefore,  in  the  development  of  this  S  JT  we  relied  heavily  upon  input  from  SMEs  to  tell  us 
about  the  appropriate  kind  and  amount  of  detail  for  each  situation. 

Generating  Response  Options 

Another  key  measurement  consideration  in  the  development  of  SJTs  is  how  to  generate 
response  options.  Although  response  options  can,  in  some  cases,  be  generated  by  the  SJT 
developer  alone,  it  is  often  prudent  to  seek  input  from  SMEs.  This  increases  the  likelihood  of 
gathering  more  responses  for  each  item  stem.  Eurthermore,  the  issues  of  response  complexity 
and  fidelity  apply  equally  to  response  options  as  to  item  stems,  and  SMEs  can  help  ensure 
response  options  are  straightforward  to  comprehend  as  well  as  realistic. 

Once  a  set  of  situations  was  written  and  edited  for  the  current  SJT,  we  conducted  a 
second  round  of  focus  groups  with  SMEs.  The  purpose  of  these  focus  groups  was  to  generate 
response  options  for  the  three  questions  associated  with  each  situational  stem.  Because  we 
developed  these  questions  based  on  a  review  of  the  literature  and  our  subsequent  definition  of 
MJP,  it  was  necessary  to  use  a  targeted  protocol  for  soliciting  information  to  populate  response 
options. 

During  the  focus  groups,  the  research  team  presented  SMEs  with  situations  to  which  they 
provided  the  following  written  responses:  (a)  describe  how  they  would  handle  the  situation, 
including  sufficient  rationale  for  their  approach,  (b)  describe  3-5  plausible  courses  of  action  that 
could  be  taken  in  response  to  the  situation,  including  both  effective  and  ineffective  actions,  (c) 
describe  1-2  possible  outcomes  or  effects  for  each  course  of  action,  and  (d)  describe  key  factors 
taken  into  consideration  in  making  the  original  decision,  including  factors  a  novice  might  weigh 
too  heavily.  Each  of  these  questions  was  asked  in  order  to  provide  information  that  could  be  used 
later  to  generate  response  options. 

Eor  the  example  situation  presented  above,  three  sets  of  response  options  were  generated, 
one  each  for  courses  of  action,  situational  factors  taken  into  consideration,  and  plausible 
outcomes.  They  are  shown  here  without  reference  to  specific  response  instructions,  which  will  be 
discussed  next. 

Courses  of  action: 

A.  Continue  with  mission  as  recommended  by  your  team  leader,  do  not  inform  HN 
forces  of  specifics  or  of  your  objective. 

B.  Tell  HN forces  to  cancel  all  of  their  patrols  for  the  next  24  hours;  tell  them  you 
will  be  conducting  a  solo  mission  and  they  are  to  stay  put. 

C.  Tell  HN  forces  that  you  will  be  conducting  a  joint  operation,  send  their  patrols  in 
a  different  direction,  clearing  the  way  for  your  team. 

D.  Object  to  your  Captain ’s  orders.  Report  to  the  Battalion  Commander  your  plans 
and  ask  him  to  override  the  Captain ’s  changes  to  your  plan. 
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E.  Change  the  plan  such  that  you  inform  HN forces  of  the  imminent  suicide  attack 
and  have  them  lay  out  the  ambush  instead  of  your  forces. 

F.  Conduct  a  joint  mission  with  HN  personnel;  confiscate  their  communication 
equipment  before  you  brief  them  on  mission  specifics. 

Situational  factors  taken  into  consideration: 

A.  Possibility  of friendly  on  friendly  fire  without  coordination. 

B.  Compromising  the  operational  security  of  the  mission. 

C.  Rapport  with  HN forces  if  they  feel  we  don ’t  trust  them. 

D.  Captain ’s  reasons  for  not  trusting  your  plan  as  you  presented  it. 

E.  Following  your  Commander ’s  advice. 

F.  Preventing  an  imminent  attack  on  the  FOB  at  all  costs. 

Plausible  outcomes: 

A.  Suicide  bomber  is  killed;  HN  forces  feel  that  you  do  not  trust  them  and  refuse  to 
work  with  you  any  further. 

B.  Your  patrol  encounters  a  HN  patrol  en  route;  HN  patrols  fires  at  you  and 
friendly-on-friendly  firefight  ensues. 

C.  Operation  is  compromised  when  a  HN  Soldier  warns  the  suicide  bomber  of  the 
impending  ambush. 

D.  Your  Captain  reprimands  you  for  not  following  his  orders;  he  states  that  one 
bomber  is  not  worth  the  risk  to  the  team. 

Each  set  of  responses  for  each  situational  stem  presents  a  representative,  but  not 
necessarily  exhaustive,  list  of  options  related  to  courses  of  action,  situational  factors  taken  into 
consideration,  and  plausible  outcomes.  We  sought  to  generate  responses  based  on  SME  input 
that  are  as  neutral  as  possible  (i.e.,  equally  attractive),  while  at  the  same  time  including  options 
representing  both  effective  and  ineffective  manifestations  of  MJP.  Lastly,  in  selecting  options  for 
situational  factors  and  plausible  outcomes,  we  attempted  to  include  options  that  seemed  feasible 
given  the  courses  of  action  initially  presented.  As  such,  other  situational  factors  and  outcomes 
could  conceivably  come  into  play  as  someone  judges  how  to  handle  the  situation.  However,  the 
constraints  of  the  test  format  necessarily  limit  the  number  of  options  that  can  be  presented. 

Choosing  Response  Instructions 

Having  considered  various  SJT  stem  and  response  generation  methods,  as  well  as  the 
level  of  complexity  in  SJT  stems  and  responses,  it  is  important  to  also  consider  the  issue  of 
response  instructions.  The  key  issue  here  is  deciding  whether  to  use  a  behavioral  tendency 
format  in  which  the  respondent  is  asked  to  indicate  what  he  would  do  in  a  situation,  or  use  a 
knowledge  format  in  which  the  examinee  is  asked  to  indicate  what  he  should  do  in  a  situation.  It 
is  possible  that  the  criterion-related  validities  of  the  SJT  will  vary  depending  on  the  format  used. 
While  Ployhart  and  Ehrhart  (2003)  found  that  SJTs  with  would  do  instructions  had  higher 
criterion-related  validities,  McDaniel  et  ah,  (2001)  found  higher  criterion-related  validities  for 
should  do  instructions  in  their  meta-analysis  of  SJT  validities.  Recent  evidence  indicates  that  the 
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behavioral  tendency  approach  is  more  likely  to  capture  typical  performance  and  correlate  more 
highly  with  measures  of  personality;  whereas,  knowledge  SJTs,  considered  a  maximal 
performance  measure,  correlated  more  highly  with  measures  of  cognitive  ability  (McDaniel, 
Hartman,  Whetzel,  &  Grubb,  2007).  Because  we  wish  to  capture  both  elements  in  this  measure, 
we  elected  to  include  questions  eliciting  both  knowledge  and  behavioral  tendency. 

First,  we  will  ask  participants  to  rate  the  effectiveness  of  each  of  the  options  representing 
the  plausible  courses  of  action  on  a  7-point  scale  from  1,  very  ineffective,  to  7,  very  effective. 

This  response  instruction  format  generates  the  largest  amount  of  information,  and  solicits 
responses  based  on  participant  knowledge.  Two  additional  questions  based  on  a  behavioral 
tendency  approach  ask  participants  to  rate  which  of  all  of  the  plausible  courses  of  action  they 
would  be  most  and  least  likely  to  take.  There  is  evidence  to  suggest  that  asking  about  what  the 
individual  is  “most  likely”  versus  “least  likely”  to  do  captures  different  aspects  of  the  judgment 
construct  (e.g.,  Cucina,  Vasilopoulos,  &  Teaman,  2003;  McElreath  &  Vasilopoulos,  2002; 
Vasilopoulos,  Goldenberg,  Cucina,  Hayes,  &  McElreath,  2005).  Therefore,  we  chose  to  capture 
both  of  these  pieces  of  information  in  our  measure  by  asking  participants  to  answer  the  following 
questions: 


Which  of  the  above  courses  of  action  (COA)  would  you  be  most  likely  to  take  (circle 
only  one  option)? 

Which  of  the  above  COA  would  you  be  least  likely  to  take  (circle  only  one  option)? 

This  first  multiple  choice  question  about  courses  of  action  most  closely  resembles  a 
traditional  S  JT  item  in  terms  of  its  response  instruction  format.  S  JT  measures  typically,  however, 
ask  for  ratings  from  either  a  knowledge  or  a  behavioral  tendency  approach.  Yet  several  recent 
calls  in  the  literature  have  asked  SJT  researchers  to  include  both  types  of  response  format  to 
better  understand  how  response  endorsements  relate  to  important  criteria  (e.g.,  job  performance, 
training  performance)  and  other  variables  of  interest  (McDaniel,  et  ah,  2007;  Ployhart  &  Ehrhart, 
2003).  Using  both  response  formats  here  (knowledge  and  behavioral  tendency)  will  enable  us  to 
examine  their  predictive  relationship  with  key  criteria,  including  their  contribution  relative  to 
other  predictors  such  as  intelligence  and  personality. 

The  second  multiple-choice  item  will  ask  about  critical  cues  and  situational  factors. 
Participants  will  be  provided  with  a  list  of  environmental  cues/factors  present  in  the  situation  and 
will  be  asked  to  rate  each  cue/factor  for  its  importance.  This  will  allow  the  research  team  to 
discern  which  factors  were  perceived  by  participants  as  most  and  least  important  in  each 
situation.  The  response  instructions  for  this  question  ask  participants  to  rate  the  importance  of 
each  factor  on  a  7-point  scale  from  1,  highly  unimportant,  to  7,  highly  important.  It  is  presently 
unclear  whether  rating  the  importance  of  situational  factors  relies  more  on  knowledge  or 
behavioral  tendency.  Given,  however,  that  the  response  options  themselves  describe  aspects  of 
the  situation,  it  is  likely  that  ratings  of  their  importance  rely  to  a  greater  extent  on  knowledge. 

The  last  multiple-choice  question  will  assess  the  capacity  of  the  participant  to  forecast  the 
consequences  and  outcomes  of  different  courses  of  action  by  asking  him  to  provide  likelihood 
ratings.  Participants  will  be  asked  to  indicate,  from  a  list  of  plausible  outcomes  related  to  the 
situation,  what  they  believe  will  be  the  probable  outcome  of  choosing  the  most  and  least  likely 
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courses  of  action  selected  previously.  Unlike  the  other  two  multiple  ehoice  questions,  which  ask 
for  a  rating  to  be  given  for  eaeh  eourse  of  aetion  and  situational  faetor,  we  will  ask  partieipants  to 
answer  only  the  following  questions: 

Probable  outcome  of  the  COA  you  are  most  likely  to  take: _ (mark  one  A-F) 

Probable  outcome  of  the  COA  you  are  least  likely  to  take: _ (mark  one  A-F) 

Traditional  and  modified  SJT  format  type  questions  will  elieit  responses  for  three  of  the 
four  elements  of  MJP:  eourses  of  aetion,  situational  faetors,  and  plausible  outeomes.  In  order  to 
capture  information  about  problem  framing,  we  will  include  an  open-ended  question  prompting 
free  responses  from  participants.  This  open-ended  question  will  ask  partieipants  to  explain  in  1-2 
sentenees  what  they  believe  is  the  problem  or  issue  to  be  resolved  in  the  situational  stem.  Rather 
than  providing  partieipants  with  options  from  whieh  to  ehoose,  we  believe  that  it  is  more 
appropriate  to  elieit  open  responses  about  how  participants  frame  the  situational  problem  at  hand. 
Further,  using  this  format  for  one  of  the  four  questions  per  stem  allows  us  to  explore  its 
feasibility  for  future  versions  of  this  measure,  and  whether  it  provides  useful  information  relative 
to  the  SJT  format  questions. 

One  final  issue  is  to  deeide  on  a  proeedure  for  determining  the  effeetiveness  of  SJT 
responses.  At  least  three  different  methods  are  available,  ineluding  SME  seoring,  empirieal 
keying,  and  relianee  on  theory  (Weekley,  Ployhart,  &  Holtz,  2006).  Researeh  has  eompared  these 
three  methods  extensively.  The  general  conclusion  is  that  empirically-devised  and  rational  keys 
are  roughly  equivalent  in  terms  of  validity.  For  the  current  SJT,  our  plan  is  to  gather  expert 
ratings  for  eaeh  of  the  SJT  responses  from  a  group  of  SMEs. 

Through  the  questions  and  methods  deseribed,  we  believe  we  will  eapture  eritieal 
information  regarding  individuals’  eapabilities  to  (a)  aeeurately  frame  deeision  problems,  (b) 
deteet  and  interpret  important  situational  faetors/eues,  (e)  foreeast  the  eonsequenees  of  different 
eourses  of  action,  and  (d)  choose  an  effective  course  of  action  based  on  the  preeeding  three 
faetors.  In  doing  this  we  hope  to  eapture  the  eritieal  elements  of  MJP. 

Summary 

Our  goal  in  writing  this  report  was  twofold.  First,  we  sought  to  define  and  deseribe  a  skill 
we  refer  to  as  “military  judgment  profieieney,”  whieh  is  judgment  and  deeision-making  in 
environments  eharaeterized  by  eultural,  legal/ethieal,  and  taetieal  eomplexity.  We  reviewed 
relevant  literature  in  the  areas  of  judgment,  decision-making,  and  problem-solving,  whieh  laid 
the  theoretical  foundation  for  our  definition  of  the  MJP  eonstruct.  Based  on  relevant  pieees  of  the 
extant  literature,  we  defined  MJP  as  a  eomplex  skill  that  involves:  (a)  aeeurately  framing  decision 
problems,  (b)  deteeting  and  interpreting  important  situational  faetors  and  ones,  (e)  foreeasting  the 
eonsequenees  of  different  eourses  of  aetion,  and  (d)  choosing  an  effective  eourse  of  aetion  based 
on  the  preceding  three  factors.  We  argued  that  in  ambiguous,  novel,  rapidly  ehanging  situations 
in  whieh  there  is  limited  time,  information,  and  resourees.  Soldiers  demonstrating  MJP  are  more 
likely  to  seleet  an  effeetive  eourse  of  aetion  by  appropriately  identifying  the  nature  of  the 
situation,  recognizing  relevant  situational  faetors,  and  foreeasting  the  best  overall  outeome(s) 
from  the  seleeted  course  of  action,  given  the  situation.  In  eombination,  these  four  eomponents  of 
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MJP  enable  Soldiers  to  achieve  the  best  possible  end-state  when  making  decisions  in  highly 
ambiguous,  dynamic,  and  complex  situations. 

Throughout  the  literature  review  section  of  this  report,  we  revisited  the  notion  that  while 
people  are  limited  by  their  capacity  to  search  for  and  process  information,  they  are  nevertheless 
cognitively  adaptive  to  the  task  environment.  This  theme  is  central  to  how  we  conceive  of  MJP, 
and  was  discussed  in  relation  to  several  models  of  judgment  and  decision-making,  including 
rational  models,  the  Heuristics  and  Biases  (HB)  program  initiated  by  Kahneman  and  Tversky, 
Hammond’s  Cognitive  Continuum  Theory  (CCT),  the  Adaptive  Toolbox  (AT)  approach 
proposed  by  Gigerenzer  and  colleagues,  and  Klein  and  colleagues  exploration  of  Naturalistic 
Decision-making  (NDM). 

In  our  view,  based  on  a  comprehensive  literature  review.  Soldiers  make  effective 
decisions  by  appropriately  framing  the  situation  and  recognizing  relevant  situational  factors, 
which  enables  them  to  select  a  course  of  action  that  is  most  likely  to  result  in  their  desired 
outcome.  In  novel,  ambiguous,  rapidly  changing  situations  in  which  there  is  limited  time, 
information,  and  resources,  MJP  is  primarily  intuitive,  in  that  it  relies  on  domain-specific, 
adaptive  heuristics  and  well-organized  knowledge  to  arrive  at  judgments  quickly.  If  time  and 
circumstances  permit,  this  initial  intuitive  approach  can  be  endorsed,  corrected,  or  overridden  by 
a  more  deliberate,  analytical  approach. 

A  second  goal  of  this  report  was  to  describe  the  development  of  an  assessment  tool  that 
will  distinguish  among  Soldiers  on  the  MJP  construct.  We  described  the  situational  judgment 
basis  of  this  measure,  which  relies  on  both  traditional  and  modified  SJT  format  questions.  We 
described  how  the  measure  uses  multiple  formats  to  tap  each  of  the  components  of  MJP.  Moving 
beyond  theory  to  operationalizing  this  construct  is  an  essential  step  toward  establishing  its 
nomological  network.  Moreover,  from  an  applied  perspective,  being  able  to  assess  MJP  can  aid 
in  the  selection  of  Soldiers  for  specialized  units  or  teams,  such  as  Provincial  Reconstruction 
Teams  or  Special  Operations  units,  and  can  serve  as  the  basis  for  the  development  of  training 
that  targets  MJP.  Based  on  this  methodology  we  developed  an  assessment  tool  with  19 
situational  items  -  8  tactical,  6  cultural,  and  5  legal/ethical.  In  the  next  phase  of  this  work  we  will 
develop  a  scoring  procedure  and  embark  on  validation  research,  evaluating  the  reliability, 
validity,  and  other  psychometric  properties  of  the  instrument,  and  determine  the  potential 
usefulness  of  the  measure  in  applied  settings. 
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